simonlammmm/exorcise repository overview

⁠Exorcise

Exome-guided reannotation of nucleotide sequences, and the CRISPR screen analysis toolchain that goes with it. One image, five commands.

Source and full documentation: https://github.com/SimonLammmm/exorcise⁠. Runs on x86_64 and Apple silicon.

Exorcise is described in Lam, S., Thomas, J.C. & Jackson, S.P. Genome-aware annotation of CRISPR guides validates targets in variant cell lines and enhances discovery in screens. Genome Med 16, 139 (2024). https://doi.org/10.1186/s13073-024-01414-4⁠. Please cite it if you use Exorcise in your work.

You can also try Exorcise on the web at https://sjlab.cruk.cam.ac.uk/app/ddrcs/⁠ > Exorcise.

⁠Introduction

Sequences are usually annotated against one particular genome assembly, and it is not always right to carry those annotations over to another use. A library of sgRNAs for CRISPR-Cas9 might have been designed against GRCh37, with each guide described as targeting some gene or as non-targeting. Searching for those same guides in GRCh38 will turn up discrepancies, because the assembly has moved on; and a cell line that does not faithfully reflect GRCh37 will respond to the guides in ways its annotations do not predict.

Exorcise reannotates sequences according to whether they are present in a genome and exome that you supply. If that genome and exome reflect the material you are actually working with, you can be confident about which sequences are present in it and which annotations hold.

It works by aligning the sequences to the genome with BLAT, then using set operations between the alignment coordinates and the exon coordinates to transfer annotations onto the perfect alignments. Each sequence is considered independently, so a guide that turns out to cut two genes is reported against both.

Exorcise is written in R and developed by Dr Simon Lam.

⁠Quick start

Pick a tag from the Tags tab and pull it. Both architectures are published under one tag as a manifest list, so a plain pull fetches whichever matches your machine.

docker pull simonlammmm/exorcise:<tag>

Bind mount a working directory at /data and name the command you want:

docker run --rm -v .:/data/ simonlammmm/exorcise:<tag> exorcise [arguments]

Run the image with no command, or with --help, to list what it accepts.

With Singularity:

singularity pull docker://simonlammmm/exorcise:<tag>
singularity run -B .:/data/ exorcise_<tag>.sif exorcise [arguments]

Ask your cluster administrator which host directories are mounted by default and whether you may bind others with -B.

Every example below shows the command on its own. Prefix each with docker run --rm -v .:/data/ simonlammmm/exorcise:<tag>, and give paths relative to the directory you mounted.

⁠What is in the image

The repository's example/ directory is not in the image. Clone the repository if you want the example inputs, the analysis workbook template, or example/examples.txt, which works through every step as a runnable script.

You supply the genome yourself, in 2bit format:

curl -L -o hg38.2bit "https://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/hg38.2bit"

⁠The commands

Exorcise itself reannotates sequences. Four other commands cover the rest of a screen analysis, and all five are in this image.

A typical run goes exorcise-trace to find the guide region in your reads, exorcise to reannotate the library, exorcise-count to count against it, exorcise-analyse to score the screen, then exorcise-database to collect the results.

The Python commands were previously the separate crispr_tools⁠ and crispr_screen_viewer⁠ projects. They are now one package.

⁠Deprecated names

The commands were renamed in 3.0.0. Every old name still works and behaves identically; each prints a warning naming its replacement.

crispr-screen-viewer accepts database and remove here. Its launch, genes and test subcommands went with the Dash web viewer, which is not part of this project.

⁠exorcise

exorcise --infile library.tsv.gz --outdir exorcise-output/ \
  --seq 7 --harm 2 --pam NGG --mode ko \
  --genome hg38.2bit --exome /exorcise/data/hsa.grch38.refseqall.tsv.gz

Files may be plain or gzipped.

⁠CRISPR chemistry

Set the chemistry with --mode. It decides which features a guide is considered to act on.

For a custom base editor, X is the original base and Y the edited base, both in [ACGT]; nn and mm are the first and last positions of the editing window, two digits each, counting from the PAM-distal end of the guide. So beCT0208 is equivalent to cbe, and beAG0409 to abe.

Base editor modes need an exome that carries CDS coordinates and exon frames, which means the name, cdsStart, cdsEnd and exonFrames columns as well as the usual ones. The bundled exome has only five columns, so download a fuller refGene table from the UCSC Table Browser for base editing.

⁠Controls

Guides that fail to align inside any exon are named exo_Non-targeting_1, exo_Non-targeting_2 and so on. If you know that some of them are declared controls, pass --harm along with --control and --control_type and they will be named after the control they belong to instead. For example:

--harm 2 --control "NO_CURRENT,^NonTargeting" --control_type "Unmapped,NonTargeting"

names every unannotated guide whose column 2 value matches /NO_CURRENT/ as Unmapped_1, Unmapped_2 and so on. Patterns are tried in the order given, and each guide takes the first that matches.

⁠Expression masking

--expression takes a two-column file of gene symbol and expression value. Guides whose --harm annotation names a gene scoring below --exprcutoff are dropped before alignment, which is a fast way to restrict a screen to the genes a particular cell line actually expresses. It requires --harm, since that is the column the symbols are matched against.

⁠Outputs

Exorcise writes exorcise.tsv plus the intermediate files each stage produced. The intermediates are safe to delete once you are happy with the result.

Each stage is checkpointed: if its output is already present and non-empty, the stage is skipped. That makes it cheap to rerun with a different --mode against the same genome, since the BLAT alignment is reused. It also means that if you change --infile you must either use a fresh --outdir or delete the intermediates, or the stale alignment will be picked up. Exorcise warns when it notices alignments that are not in the current input.

In exorcise.tsv, Exorcise's own columns are prefixed exo_ and come first. Every column from your input file follows, unchanged. Columns inherited from the exome are prefixed inherit..

One input sequence can produce several rows: a sequence aligning to two loci is two reannotations, and in base editor mode each predicted protein change is its own reannotation.

Base editor modes add these columns:

⁠exorcise-trace

Plots the proportion of each nucleotide called at each sequencing cycle in a FASTQ file, which is a quick way to find where the fixed region of an amplicon ends and the variable guide region begins. Use the answer as --slice for exorcise-count.

exorcise-trace --file reads.fastq.gz --nrows 10000 --out trace-output/

It writes a PDF, an interactive HTML plot and a TSV of the counts per file.

⁠exorcise-count

Counts the unique sequences in the sliced region of each read, then maps them onto a guide library. Output is one dereplicated count file per sample, plus a single counts table when --library is given.

exorcise-count --slice 0,20 --prefix counts/run \
  --library exorcise-output/exorcise.tsv \
  --seqhdr exo_seq --guidehdr exo_id --genehdr exo_symbol \
  reads/*.fastq.gz

An Exorcise output makes a good library here: pass --seqhdr exo_seq --guidehdr exo_id --genehdr exo_symbol and your counts arrive already carrying genome-aware annotations.

Guides shorter than 16 bases are not counted, because below that a sequence stops being specific enough to identify one guide. If the library's guides vary in length, or are shorter than the slice, a window is slid along the sliced region.

⁠exorcise-analyse

Runs MAGeCK, DrugZ or Chronos over a screen. What runs is decided by the Analyses sheet of an .xlsx workbook; start from example/example-analysis-workbook-template.xlsx in the repository. A .json configuration carrying sample_reps, analyses and control_groups also works.

exorcise-analyse --counts counts/ --output-dir results/ my-screen.xlsx

Every analysis is checkpointed on its output file, so a rerun only does what is missing. --dry-run is worth using first: it reports every problem it can find in the workbook at once.

Chronos only applies to comparisons that span more than one timepoint with the control at the earliest one, so it needs Days grown and Trajectory columns in Sample details. Comparisons that do not qualify are skipped with a reason.

⁠exorcise-database

Collects the tables exorcise-analyse wrote into a queryable database, alongside two gzipped CSVs of experiment and comparison metadata.

exorcise-database --out-dir db/ --results-dir results/ --counts-dir counts/ \
  --new-db my-screen.xlsx

It writes database.db, experiments_metadata.csv.gz and comparisons_metadata.csv.gz. Nothing here touches the network.

--new-db over a database that already exists asks before replacing it. Under docker run there is no terminal to answer with unless you pass -it, so in a script or a scheduled job add --force-overwrite. Without either, it refuses and leaves the database alone rather than proceeding. Omit --new-db altogether to add to the existing database instead of replacing it.

⁠Removing an experiment

--remove takes an experiment out of a database. Name it either by the workbook you added it with, or by its experiment ID:

exorcise-database --remove --out-dir db/ my-screen.xlsx
exorcise-database --remove --out-dir db/ my-experiment-id

Anything ending .xlsx is read as a workbook and its "Experiment name" is used; anything else is taken as an ID literally. The IDs in a database are listed in its experiments_metadata.csv.gz. Several may be given at once.

This removes the experiment's statistics and its rows from both metadata tables. Gene records stay, because they are shared between experiments. Experiments you name that are not in the database are reported and skipped, not treated as errors.

Like --new-db, removal asks first, needs -it under docker run to be answerable, and takes --force-overwrite to proceed unattended.

⁠Indexing, and CRAVE

A finished database is indexed for the queries downstream tools make. The stat table's primary key is ordered (comparison_id, gene_id, analysis_type_id), so it only helps queries that filter on comparison first; anything asking about one gene across many comparisons would otherwise scan the whole table. Two indexes are added once the rows are written:

This happens automatically whenever a database is created, added to or updated, so normally there is nothing to do. For a database built before this was added:

exorcise-database --reindex --out-dir db/

which creates whatever is missing, runs ANALYZE, and changes no data. It is idempotent, so it is safe to run on a database that is already indexed.

The index names are CRAVE⁠'s. The output directory of exorcise-database is a CRAVE dataset directory: point an instance of CRAVE at it and you get within- and between-screen exploration over the results, scored against genome-aware annotations all the way back to the library. CRAVE creates these indexes itself if they are absent, looking for exactly these names, so matching them means it finds them already there instead of spending minutes rebuilding them at startup — and it cannot build them at all when the dataset directory is mounted read-only, as it usually is in a deployment.

⁠Other ways to install

Conda environments and a source install are both supported, and the image builds from a working tree if you want to test a change. See the README on GitHub⁠ for both, and for the worked examples in example/examples.txt.

git clone https://github.com/SimonLammmm/exorcise
cd exorcise
docker build -f docker/Dockerfile -t simonlammmm/exorcise .

The build context is the repository root, which is why -f is needed and why the trailing . matters.

⁠About

Exorcise is developed and maintained by Dr Simon Lam, University of Cambridge, [email protected]⁠.

exorcise-count, exorcise-analyse and exorcise-database derive from crispr_tools and crispr_screen_viewer by John C. Thomas. exorcise-analyse bundles DrugZ⁠ by Medina Colic and Traver Hart, whose numerical routines are reproduced unchanged.

⁠Licence

This code is licensed under the Creative Commons Zero v1.0 Universal licence.

Exorcise uses BLAT⁠, which is free for academic, personal and non-profit use. exorcise-analyse calls MAGeCK⁠ and, optionally, Chronos⁠; both carry their own licences.