Harmonization and Integration Pipeline for Functional Genomics. bitbucket.com/wanglab-upenn/hipfg
177
The Harmonization and Integration Pipeline for Functional Genomics (hipFG) is a tool to normalize data and metadata of diverse functional genomics formats.
For a more detailed explanation, please see the manuscript published in Bioinformatics here.
To use hipFG in its Docker container, please pull from the Docker hub.
hipFG is run by executing hipFG.sh and specifying, at minimum --mdt to indicate the MDT:
bash <your path>/hipFG.sh --mdt mdt.tsv
Additional options are available:
Usage:
bash hipFG.sh --mdt <your TSV>
options:
--mdt A TSV pointing to input files, output and temporary directories, and input metadata.
--run_parallel Optional. true/false. Whether to run multiple samples at a time. Default false.
--num_jobs Optional. Integer. How many samples to run at a time. Default 0/non-parallel.
--testing Optional. true/false. If true, only write code for the first two samples. Default false.
--run_now Optional. true/false. If true, run auto-generated scripts immediately. This may initiate parallel run if specified.
--metadata_only Optional. true/false. If true, re-generate sample scripts but only execute metadata portions. Useful when MDT is updated but output data does not need to be re-generated. Default false.
--replace_metadata Optional. true/false. Default behavior is for newest generated metadata to be appended to project-level metadata. If true, do not append to a master output metadata, and instead replace it. Use caution when calling! Default false.
--simple_mdt Optional. true/false. If true, hipFG requires only 7 fields in MDT but skips metadata steps. Default false (20 required fields).
hipFG is available via the Docker hub.
docker pull wanglab/hipfg
Use the Docker container to apply to new data or to run the built-in hipFG examples.
The Docker container can be applied to external data by:
hipFG.sh with desired options.For example, following the docker pull, interval annotations can be normalized via hipFG using local directories. All necessary directories are mounted using the -v option:
docker run -v /home/jcifello/input_data:/home/jcifello/input_data \
-v /home/jcifello/test_docker/my_output:/home/jcifello/test_docker/my_output \
-v /home/jcifello/test_docker/my_temp:/home/jcifello/test_docker/my_temp \
-v /home/jcifello/test_docker/project_dir:/home/jcifello/test_docker/project_dir \
-w /app wanglab/hipfg bash /app/hipFG/hipFG.sh --mdt /home/jcifello/test_docker/project_dir/mdt.tsv --run_now true
And QTLs can be executed similarly, noting an additional mounted directory for the reference files:
docker run -v /home/jcifello/input_qtls:/home/jcifello/input_qtls \
-v /home/jcifello/test_docker/my_output:/home/jcifello/test_docker/my_output \
-v /home/jcifello/test_docker/my_temp:/home/jcifello/test_docker/my_temp \
-v /home/jcifello/test_docker/project_dir:/home/jcifello/test_docker/project_dir \
-v /home/jcifello/hipfg_ref_files:/home/jcifello/hipfg_ref_files \
-w /app wanglab/hipfg bash /app/hipFG/hipFG.sh --mdt /home/jcifello/test_docker/project_dir/mdt_qtl.tsv --run_now true
As a template, define the following variables for your system, and execute the docker command below:
input_dir=</abs/path/>
output_dir=</abs/path/>
project_dir=</abs/path/>
docker run -v $input_dir:$input_dir \
-v $output_dir:$output_dir \
-v $project_dir:$project_dir \
-w /app wanglab/hipfg bash /app/hipFG/hipFG.sh --mdt $project_dir/mdt.tsv --run_now true
NOTE: Any paths provided in the MDT must be absolute and downstream of a mounted directory.
For application to QTLs, carry this out with an additional mount for $qtl_references_dir.
To practice running the Docker container on external data, we can use files available in this repo. Download the xQTL related files here:
## download the input data
wget https://bitbucket.org/wanglab-upenn/hipfg/raw/9984f730fb6486133348ec3e233eb2128aa009cf/examples/xQTLs/inputs/Alasoo_2018_ge_macrophage_IFNg.all.tsv.gz
wget https://bitbucket.org/wanglab-upenn/hipfg/raw/9984f730fb6486133348ec3e233eb2128aa009cf/examples/xQTLs/inputs/GTEx_ge_brain_spinal_cord.all.tsv.gz
## download data descriptors
wget https://bitbucket.org/wanglab-upenn/hipfg/raw/9984f730fb6486133348ec3e233eb2128aa009cf/examples/xQTLs/config.txt
wget https://bitbucket.org/wanglab-upenn/hipfg/raw/9984f730fb6486133348ec3e233eb2128aa009cf/examples/xQTLs/mdt.tsv
## Init Config
wget https://bitbucket.org/wanglab-upenn/hipfg/raw/be41a5dcce4e5726e4a457c9f5876438a80f1996/hipFG.ini
Relocate the input files to an input directory of your choice. Update the following fields in this MDT to point to the recent downloads:
Note: All paths must be absolute when using the Docker container.
Lastly, update these paths of the new .ini file:
dbSNP_hg38=/app/hipFG/examples/xQTLs/reference/dbSNP_custom_example_reference.txt
hgnc_gencode_hg38=/app/hipFG/examples/xQTLs/reference/hg38_reference_genes.txt
reference_genome_hg38=/app/hipFG/examples/xQTLs/reference/hg38_short_reference_genome.fa
With the absolute paths used in the MDT, mount the necessary (absolute) paths and run hipFG via the Docker:
input_dir=</abs/path/>
output_dir=</abs/path/>
project_dir=</abs/path/>
log_dir=</abs/path/>
docker run -v $input_dir:$input_dir \
-v $output_dir:$output_dir \
-v $project_dir:$project_dir \
-v $log_dir:$log_dir \
-w /app wanglab/hipfg bash /app/hipFG/hipFG.sh --mdt $project_dir/mdt.tsv --run_now true --log_dir $log_dir --force_overwrite true
The examples carried out to verify successful install above can also be executed within the Docker container:
docker run -w /app wanglab/hipfg sh -c "cd /app/hipFG/examples/intervals/ && bash run_me.sh"
docker run -w /app wanglab/hipfg sh -c "cd /app/hipFG/examples/interactions/ && bash run_me.sh"
docker run -w /app wanglab/hipfg sh -c "cd /app/hipFG/examples/xQTLs/ && bash run_me.sh"
Content type
Image
Digest
sha256:dd20300f4…
Size
555.4 MB
Last updated
almost 3 years ago
docker pull wanglab/hipfg