ultimagenomics/sorter repository overview

⁠Overview

Demux takes an input SAM file and generates a directory with a file per barcode. It also includes metadata and statistics. Sorter takes one or two directories created by Demux. It performs: data-merging (between the two directories), sorting, mark-duplicates, statistics collection, and sampling. It generates a top-level directory with a sub-directory per barcode. We use the terms read-group and barcode interchangeably here.

The two programs work together; demux creates metadata that sorter requires. Sorter cannot be run on its own.

⁠A little bit about the implementation

Demux uses a single thread to read from the input stream. This is the reader thread. It reads in big chunks, splits into SAM lines, and adds the records to a per-barcode memory-buffer. A pool of worker threads waits on a shared queue of buffers. Once a buffer fills up, the first available worker picks it up, compresses, and writes to the appropriate barcode file.

Sorter aims for maximal parallelism. It tries to use all of the threads and memory available to it. For example, statistics are calculated by: (1) splitting the data into chunks, (2) collecting statistics per chunk, (3) summing up the statistics per read-group. The strategy is to collect histograms that have an additive property, making it easy to sum them.

Duplicate Marking is implemented as part of the sorter.

⁠Get sorter Docker Image

⁠Get the Image

The docker image for UA can be downloaded with

$ docker pull ultimagenomics/sorter

and given a shorter name with

$ docker tag ultimagenomics/sorter sorter

This shorter name is used in the following section.

⁠Run the Container

The sorter and demux executables are in the PATH of the docker image, so the simplest invocation would be

$ docker run --rm sorter sorter --version

5.2.0 292d8a0+ 2024-02-08 13:43:30+0000


$ docker run --rm sorter demux --help

        ...
        --dry-run                               Check that the configuration parameters are legal. Do not process the data nor generate output.
        --fastq-underscores-for-missing-fields  When generating FASTQ use underscores to stand in for missing fields
        --help                                  Print options.
        --cram-ignore-md5                       ignore md5 errors in CRAM files
        --input=<file>                          Input SAM file.  May be "-" for stdin.  Default is stdin.
        ...

The complete README.md file can be found at README.md in the working directory of the container and can be viewed with a command-line markdown viewer such as mdless.

$ docker run --rm sorter cat README.md | mdless

Running demux, assuming the dataset is on volume data:

docker run --user $(id -u):$(id -g) --volume /data:/data --rm sorter demux \
   --input=/data/test/1583.sam \
   --output-dir=/data/test/output \
   --align=true \
   --reference /data/Reference/Homo_sapiens_assembly38.fasta \
   --runid 1583 \
   --progress

Running sorter on the demux output:

mkdir /data/test/sorter_output
docker run --user $(id -u):$(id -g) --volume /data:/data --rm sorter sorter \
   --input-dir=/data/test/output \
   --output-dir=/data/test/sorter_output \
   --runid 1583 \
   --progress

We need the host user-id to grant docker write permissions to the output directory. This is done with (--user $(id -u):$(id -g)).