tengfei/encode-chip-seq-pipeline repository overview

⁠Introduction

This is an effort to create a docker image that contains the whole conda-based enviroment setup to execute full the ENCODE Transcription Factor and Histone ChIP-Seq processing pipeline⁠ which implemented in WDL/Cromwell/Caper. Installation is geenrally following instruction encode⁠conda⁠

⁠Major Changes

Following changes might not be needed in the future

• Silently install miniconda3 and activate via a differenet approach for dockerfile
• libssl1.0.0 for ubuntu 18.04 install (for some unsolved bugs)
• pip install caper==0.7.0 croo==0.3.4 qc2tsv a special versino to overcome latest autouri error.
• caper init local automatically to prepare
• Udpate /root/.caper/default.conf tmp file to /root/tmp

⁠Requirements

1. Please check and edit the hg38_bip.tsv⁠ under the test folder, to make sure the absolute path is correct for all mounted files

2. Please check and edit the input job json under test folder, make sure the absolute path for your data is correct

3. WARNING: caper init local used, so only local mode now.

⁠Prepare docker runner

1. Mount datasets "ENCODE"
2. Use image "tengfei/encode-chip-seq-pipeline"
3. Instance type "r5.4xlarge" 16cpu + 128G (assuming, 2 rep *2)
4. Initial command "/bin/bash"

⁠Docker runner ready

This will create an envinronment with conda and everything ENCODE use for chip-seq analysis. User will be landing in a working folder /root/chipseq-run/. And in subfolder /root/chipseq-run/bin/ you will find our main wrapping script caper_run.sh

What it does? it's a wrapper for following steps:

1. setup environment and activate
2. run caper run
3. run croo to organize outputs
4. run qc2tsv to organize qc files into single tsv file

How to use caper_run.sh?

• it accepts one required positional argument: input job json, you will find couple /root/chipseq-run/
• it accepts second optional positional argument: output folder will be available under /enigma/local_storage/ by default, if will create a folder where you execute your command, if you execute your command in /root/chipseq-run/ directly, it will collect outputs into /enigma/local_storage/chipseq-run/. Because we will be running multiple example, so we create folder for each example and run exmaple code from there. See below.

⁠Test run example

## you should be here by default
cd /root/chipseq-run/

## make one job folder that will contain all cromwell execution result 
mkdir ENCSR000DYI_subsampled_truerep
cd ENCSR000DYI_subsampled_truerep

## execute caper_run.sh in this folder
nohup caper_run.sh /root/chipseq-run/bip_ENCSR000DYI_subsampled_truerep.json &

⁠Check outputs

The results should be collected based on metadata.json and all in /enigma/local_storage/ENCSR000DYI_subsampled_truerep/ you can archive the outputs and check results.

⁠About input JSON

Please check official Input JSON manaual⁠ as reference or a shorter version⁠ as checklist. Some JSON examples are provided (minimum⁠ and full⁠)

FOR platform maintainer, I picked some key input worth exposing

⁠Pipeline metadata

⁠Pipeline parameters

IMPORTANT if pseudo replicate generation and all related analyses caused trouble, datq quality or other problem, try change chip.true_rep_only to true

⁠Reference genome

UI could use hg38 and hg19 to automatically change following parameters, because it need to make sure genome_tsv has correct path to all files.

chip.genome_tsv| File | Choose one of the TSV files listed below or build your own

⁠Input genomic data

Pipeline can start from any of the following data type (FASTQ, BAM, NODUP_BAM and TAG-ALIGN). But you can start fro fastq

For controls:

⁠Optional mapping parameters

⁠Optional control parameters

⁠Optional peak-calling parameters

⁠Suggestion

Allow user to upload a job JSON that will overide or add additional parameters, except input files and genome_tsv (hardcoded locally for mounted files). For example, if our default job json is default.json, this will be merged and updated by user's own new.json file before execution with caper_run.sh.