cmgantwerpen/cromwell-testing repository overview

⁠CROMWELL TESTING FRAMEWORK

The goal of this repository is to provide a semi-automatic way to run multiple small WDL files on Cromwell-AWS and evaluate outputs against preset criteria. This can be used as a proxy for unit-testing, or to evaluate new cromwell releases against expected functionality.

⁠Included configurations

The repo contains:

• two test sets under /opt/cromwell-testing/tests_*:

  • callCache Referencing : re-use existing job outputs by the "reference" config setting
  • callCache Copying : S3 job outputs are copied to the new run prefix in the cromwell temp-bucket, by the "copy" config setting

• a cromwell jar under /opt/cromwell-testing/cromwell/binary

• example cromwell configs under /opt/cromwell-testing/cromwell/config Both tests sets contain tests on :

• S3 and EFS Files

• various runtime attribute settings

• callcaching

• support for globbing and optional input/output objects

⁠Dependencies

We use an in-house collection of python modules. As it contains some senstive data, the version included here is a cleaned out release. When changing code under "CMG/", please notify us so we can incorporate it upstream.

⁠SETUP

Edit the configuration files under "/opt/cromwell-testing/cromwell/configs/" to match your account.

• set aws.region and aws.profile to match ~/.aws/credentials
• set backend.providers.AWSBatch.config.root to your cromwell execution bucket : "s3://myBucket/cromwell-execution"
• set backend.providers.AWSBatch.config.aws.default-runtime-attributes.scriptBucketName to your cromwell temp bucket : "myBucket"
• set backend.providers.AWSBatch.config.aws.default-runtime-attributes.queueArn to your cromwell queue : "arn:aws:batch:eu-west-2:accountID:job-queue/myQueue"
• if not running in docker : update the mysql settings to connect to your database.

⁠Example usage:

docker run -v /home/geert/cromwell-testing:/home/geert/cromwell-testing \
           cmgantwerpen/cromwell-testing:latest \
           python /opt/cromwell-testing/WDL_Evaluator.py \
             -c /opt/cromwell-testing/cromwell/configs/configuration_functional_cacheReference.ini \
             --results /home/geert/cromwell-testing/results/

⁠outputs:

The script results are printed as a summary on how many tests were run, succeeded, and failed, and how the outputs were evaluated. All execution results are pickled in the results folder.

To visualize individual results: use:

# in python
import pickle
from pprint import pprint as pp

with open('results/results.pkl', 'rb') as f:
     data = pickle.load(f)

# print data on test 23
pp(data[23])