alexwaldrop/gap repository overview

⁠CloudConductor

⁠What is CloudConductor?

CloudConductor is a cloud-based workflow engine for defining and executing bioinformatics pipelines in a cloud environment. Currently, the framework has been tested extensively on the Google Cloud Platform⁠, but will eventually support other platforms including AWS, Azure, etc.

⁠Feature Highlights

• User-friendly
  • Define complex workflows by linking together user-defined modules that can be re-used across pipelines
  • Config_obj⁠ for clean, readable workflows (see below example)
  • +50 pre-installed modules for existing bioinformatics tools
• Portable
  • Docker integration ensures reproducible runtime environment for modules
  • Platform independent (currently supports GCS; AWS, Azure to come)
• Modular/Extensible
  • User-defined Plug-N-Play modules
    • Re-used across pipelines, re-combined in any combination
    • Modules easily added, customized as new tools needed, old tools changed
    • Eliminates copy/paste re-use of code across workflows
• Pre-Launch Type-Checking
  • Strongly-typed module declarations allow catching pipeline errors before they occur
  • Pre-launch checks make sure all external files exist before runtime
• Scalable
  • Removes resource limitations imposed by cluster-based HPCCs
• Elastic
  • VM usage automatically scales to match input file sizes, computational needs
• Scatter-Gather Parallelism
  • In-built logic for dividing large tasks into small chunks and re-combining
• Economical
  • Preemptible/Spot instances drastically cut workflow costs

⁠Setting up your system

CloudConductor is currently designed only for Linux systems. You will need to install and configure the following tools to run your pipelines on Google Cloud:

1. Python⁠ v2.7.*

   You can check your Python version by running the following command in your terminal:

   $ python -V
   Python 2.7.10
   

   Copy

   To install the correct version of Python, visit the official Python website⁠.

2. Python packages: configobj, jsonschema, requests

   You will need pip⁠ to install the above packages. After installing pip, run the following commands in your terminal:

   # Upgrade pip
   sudo pip install -U pip
   
   # Install Python modules
   sudo pip install -U configobj jsonschema requests
   

   Copy

3. Google Cloud Platform⁠ SDK

   Follow the instructions⁠ on the official Google Cloud website.

⁠Usage

For more information about CloudConductor and how to use it check the documentation⁠.

    usage: CloudConductor [-h] --input SAMPLE_SET_CONFIG --name PIPELINE_NAME
                          --pipeline_config GRAPH_CONFIG --res_kit_config
                          RES_KIT_CONFIG --plat_config PLATFORM_CONFIG --plat_name
                          PLATFORM_MODULE [-v] -o FINAL_OUTPUT_DIR
    
    optional arguments:
      -h, --help            show this help message and exit
      --input SAMPLE_SET_CONFIG
                            Path to config file containing input files and information for one or more samples.
      --name PIPELINE_NAME  Descriptive pipeline name. Will be appended to final output dir. Should be unique across runs.
      --pipeline_config GRAPH_CONFIG
                            Path to config file defining pipeline graph and tool-specific input.
      --res_kit_config RES_KIT_CONFIG
                            Path to config file defining the resources used in the pipeline.
      --plat_config PLATFORM_CONFIG
                            Path to config file defining platform where pipeline will execute.
      --plat_name PLATFORM_MODULE
                            Platform to be used. Possible values are:
                               Google (as module 'GooglePlatform')
      -v                    Increase verbosity of the program.Multiple -v's increase the verbosity level:
                               0 = Errors
                               1 = Errors + Warnings
                               2 = Errors + Warnings + Info
                               3 = Errors + Warnings + Info + Debug
      -o FINAL_OUTPUT_DIR, --output_dir FINAL_OUTPUT_DIR
                            Absolute path to the final output directory.
                            

⁠A simple pipeline example

Below, we use CloudConductor's in-built scatter-gather logic to align a set of reads to a reference genome.

# Trim input FASTQ reads with Trimmomatic
[trim_reads]
	module      = Trimmomatic

# Scatter trimmed FASTQ into smaller chunks for fast alignment
[split_fastq]
	module      = FastqSplitter
	input_from  = trim_reads

# Align chunks in parallel to reference genome using BWA
[align_reads]
	module      = BWA
	input_from  = split_fastq, get_read_group

# Index BAM files output by BWA
[index_bam]
	module      = Samtools
	submodule   = Index
	input_from  = align_reads

#  Merge split BAM files into single bam
[merge_bams]
	module      = MergeBams
	input_from  = align_reads, index_bam

⁠What Next?

Get started with our full documentation⁠ to explore the ways CloudConductor can streamline the development and execution of complex, multi-sample workflows typical in bioinformatics.

⁠Project Status

CloudConductor is actively under development. To get involved or request features, please any of the authors listed below.

⁠Authors

• Alex Waldrop⁠
• Razvan Panea⁠
• Tushar Dave⁠