dockerized loop server utilizing SGE to process various loopgenomics kits.
310
The container will be used to process various kits provided by Loop Genomics (https://www.loopgenomics.com).
Minimum System Requirements
CPU: 32 cores (64 threads)
RAM: 128G
Storage: 2G
Temporary Storage: 200G
Installation
pull the image into a local machine:
docker pull loopgenomics/loop-server:1.0.0
Run
start from obtaining the distributed LoopAnalysis package (LoopGenomics-2.1) through [email protected]:
Unzip LoopGenomics-2.1.zip including "LoopAnalysis", "jars" and "references".
Prepare "data", "temp" and "result" dirs (local volumes). For example, /mnt/temp to store intermediate files /mnt/results to store the final output files /mnt/data to store fastq files
Create a container.
LOOPGENOMICS=/path/of/LoopGenomics-2.1
TEMP=/mnt
docker run --hostname=loop-server
-v $LOOPGENOMICS/LoopAnalysis:/root/LoopAnalysis
-v $LOOPGENOMICS/jars:/root/jars
-v $LOOPGENOMICS/references:/root/references
-v $TEMP/results:/root/results
-v $TEMP/data:/root/data
-v $TEMP/temp:/root/temp
-ti loopgenomics/loop-server:1.0.0
"/path/of/LoopGenomics-2.1" is where you extracted "LoopGenomics-2.1.zip". The volumes (directories) will be linked to the directories in the container.
The "docker run" command let you log in the container. In the container, go to "/root" dir in the docker container. If you do "ls", you will see the structure of directory.
Prepare env.xml in the "/root" as shown below: 2.1 /root/temp /root/results /root cluster 24
Run pipeline
in the container, run the command to create pipeline scripts at "/root/temp/test1.
java -jar /root/jars/LoopGenomicsPipeline-2.1.jar
-env /root/env.xml \
-runName SCC1
-read1 /root/data/16s/SSC/SSC-5minSPRI_S4_L001_R1_001_subset.fq.gz
-read2 /root/data/16s/SSC/SSC-5minSPRI_S4_L001_R2_001_subset.fq.gz
-task create
-batchId 111
-kit 16S
To run the created script, replace "-task create" with "-task run".
Run 18S_ITS kit,
please download references :
Then, extract those and move to ..../LoopGenomics-2.1/reference.
You would see those reference in your docker container:
root@loop-server:~/references# ls
UNITE ensembl-fungi references.csv silva-132-SSURef silva-18S
The command would be like this (please make sure the "-kit" option) :
java -jar /root/jars/LoopGenomicsPipeline-2.1.jar -env /root/env.xml \
-read1 /root/data/LW/LW-DecQC_S1_L001_R1_001.fastq.gz \
-read2 /root/data/LW/LW-DecQC_S1_L001_R2_001.fastq.gz \
-task run -batchId 1
-kit 18S_ITS
-runName my_18S_ITS_test
Tips
The "docker run" with the options "-ti" will let you create a container and login into the container. However, when you "CTRL-D" or "exit", the container will stop itself as well as all the jobs. There is a way to avoid the inconvenience. Using the "-d" option while creating a loop-server container, the container can be running as a daemon.
For example,
docker run --hostname=loop-server \
-v $LOOPGENOMICS/LoopAnalysis:/root/LoopAnalysis \
-v $LOOPGENOMICS/jars:/root/jars \
-v $LOOPGENOMICS/references:/root/references \
-v $TEMP/results:/root/results \
-v $TEMP/data:/root/data \
-v $TEMP/temp:/root/temp \
-ti -d loopgenomics/loop-server:1.0.0
It will let the container run in the background as a daemon. You are still in the local environment, not in the container. For example,
~$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4ce7e48353c4 loopgenomics/loop-server:1.0.0 "/root/loopgenomics_…" 6 seconds ago Up 4 seconds peaceful_ritchie
You can access the container using "docker exec" without stopping the container as well as all the jobs running in the container. For example, the following shows how to login the container with its name such as "peaceful_ritchie" and how to run the LoopAnalysis package:
~$ docker exec -ti peaceful_ritchie /bin/bash
root@loop-server:/#~# java -jar /root/jars/LoopGenomicsPipeline-2.1.jar -env /root/env.xml -runName LW3 -read1 /root/data/16s/LW_DecQC_S1/LW-
DecQC_S1_L001_R1_001.fastq.gz -read2 /root/data/16s/LW_DecQC_S1/LW-DecQC_S1_L001_R2_001.fastq.gz -task run -batchId 111 -kit 16S
Kit : 16s
task : run
created ... /root/temp/LW3/scripts/demult.sh
created ... /root/temp/LW3/scripts/dictread.sh
created ... /root/temp/LW3/scripts/run.sh
created ... /root/temp/LW3/scripts/dictsplit.sh
created ... /root/temp/LW3/scripts/barcodehasher.sh
created ... /root/temp/LW3/scripts/shortreadmap_sub.sh
created ... /root/temp/LW3/scripts/dictread_sub.sh
created ... /root/temp/LW3/scripts/samplestats.sh
created ... /root/temp/LW3/scripts/classify_sub.sh
created ... /root/temp/LW3/scripts/runstats.sh
created ... /root/temp/LW3/scripts/collectout.sh
created ... /root/temp/LW3/scripts/otuclassify.sh
created ... /root/temp/LW3/scripts/classify.sh
created ... /root/temp/LW3/scripts/shortreadmap.sh
CMD : /bin/sh /root/temp/LW3/scripts/run.sh
Your job 1 ("b111demult") has been submittedYour job-array 2.1-24:1 ("b111bchash") has been submittedYour job-array 3.1-24:1 ("b111dictsplit") has been
submittedYour job-array 4.1-24:1 ("b111dictread") has been submitted
After you exit the container, you can log in again and check the status of your job submitted in the container. Here is a way to check the status of your job without logging in the container. As shown below, you will see the status of your pipeline in your local environment.
~$ docker exec -ti peaceful_ritchie qstat
job-ID prior name user state submit/start at queue slots ja-task-ID
"-----------------------------------------------------------------------------------------------------------------"
1 0.50000 b111demult root r 07/08/2019 23:36:57 all.q@loop-server 16
2 0.00000 b111bchash root hqw 07/08/2019 23:36:52 1 1-24:1
3 0.00000 b111dictsp root hqw 07/08/2019 23:36:55 1 1-24:1
4 0.00000 b111dictre root hqw 07/08/2019 23:36:58 1 1-24:1
Finally, while creating the container, the "results" dir and "temp" dir were linked between your local volume and the container's volume. You can see the final outputs in "/mnt/results" in your local environment.
Content type
Image
Digest
Size
1.4 GB
Last updated
almost 7 years ago
docker pull loopgenomics/loop-server:1.0.0