Twitcher Workbench: viewer and CLI toolkit for Twitcher template-switch aligner output.
2.9K
A browser-based viewer for Twitcher CSV output.
Load a Twitcher CSV, select a reference genome, optionally select gene annotation and repeat-masking resources, and click any row to visualize the event with Twitcher, JBrowse, IGV/JBrowse, DNA fold comparison, or raw BAM slice tools.
For the twitcher-postprocess CLI (preprocessing, labeling, filtering, group filters, and every supported filter operator), see docs/CLI.md.
read_label values for exact read_id values without modifying the original identifierssample_label values for exact sample_name values (new-version Twitcher output)read_id) and new-version (sample_name) Twitcher output formatsread_id and sample_name workflowsLarge reference files are not stored in Git and are not baked into the Docker image.
The repository keeps only the folder structure. Before running the application with bundled resources, place reference genomes, annotation files, and repeat-mask files under the ref/ directory.
The application treats files under ref/ as ready-to-use production resources. The app should not sort, normalize, bgzip, tabix-index, or otherwise rewrite bundled resources at runtime. This keeps the web viewer responsive, makes Docker/HPC usage predictable, and allows ref/ to be mounted read-only.
The reference FASTA, gene annotation GFF3, and repeat-mask GFF3 must use the same chromosome naming convention.
For example, if the FASTA index contains:
chr1
chr2
chr3
...
chrX
chrY
chrM
then the annotation and repeat-mask GFF3 files must also use:
chr1
chr2
chr3
...
chrX
chrY
chrM
If the FASTA uses chr13 but the GFF3 uses 13, JBrowse may show the gene annotation track as present but empty. The backend gene annotation code can normalize chromosome names internally, but JBrowse reads the raw bgzipped/indexed GFF3 directly, so the files in ref/ must already be compatible.
ref/ structureRecommended structure for an hg38 setup:
ref/
├── genomes/
│ └── hg38/
│ ├── hg38.no_alt.fa.gz
│ ├── hg38.no_alt.fa.gz.fai
│ └── hg38.no_alt.fa.gz.gzi
├── annotations/
│ └── hg38/
│ ├── Homo_sapiens.GRCh38.115.ucsc.sorted.gff3.gz
│ └── Homo_sapiens.GRCh38.115.ucsc.sorted.gff3.gz.tbi
└── masks/
└── hg38/
├── hg38.no_alt.masking.v2.sorted.gff3.gz
└── hg38.no_alt.masking.v2.sorted.gff3.gz.tbi
The app scans these folders and shows available bundled resources in dropdown menus. If the folders are empty, the app still works. Users can upload a custom reference genome, gene annotation GFF3, and repeat-mask GFF3 through the UI.
Install the following command-line tools before preparing reference resources:
samtools
bgzip
tabix
On Debian/Ubuntu-based systems, samtools, bgzip, and tabix are commonly available from packages such as samtools and tabix.
Place the FASTA file under:
ref/genomes/<genome_name>/
Example:
ref/genomes/hg38/hg38.no_alt.fa.gz
The FASTA must be indexed:
samtools faidx ref/genomes/hg38/hg38.no_alt.fa.gz
Expected files:
hg38.no_alt.fa.gz
hg38.no_alt.fa.gz.fai
hg38.no_alt.fa.gz.gzi
If .gzi is missing, regenerate the compressed FASTA with bgzip and index again.
Check FASTA chromosome names:
cut -f1 ref/genomes/hg38/hg38.no_alt.fa.gz.fai | head
The output defines the chromosome names that annotation and repeat-mask files must match.
Place gene annotation files under:
ref/annotations/<genome_name>/
Recommended hg38 example:
ref/annotations/hg38/Homo_sapiens.GRCh38.115.ucsc.sorted.gff3.gz
ref/annotations/hg38/Homo_sapiens.GRCh38.115.ucsc.sorted.gff3.gz.tbi
The GFF3 file must be:
bgziptabix -p gffDo not place an Ensembl-style GFF3 using 1, 2, 13, X, Y, MT into ref/annotations/hg38/ if the FASTA uses chr1, chr2, chr13, chrX, chrY, chrM. Convert it first.
chr styleUse this when the GFF3 uses Ensembl-style seqids such as 1, 2, 13, X, Y, MT, but the FASTA uses UCSC-style names such as chr1, chr2, chr13, chrX, chrY, chrM.
Create a script:
mkdir -p scripts
nano scripts/prepare_gff3_for_jbrowse.sh
Paste:
#!/usr/bin/env bash
set -euo pipefail
if [ "$#" -ne 2 ]; then
echo "Usage: $0 input.gff3[.gz] output.sorted.gff3.gz" >&2
exit 1
fi
IN_GFF="$1"
OUT_GFF="$2"
if ! command -v bgzip >/dev/null 2>&1; then
echo "ERROR: bgzip not found in PATH" >&2
exit 1
fi
if ! command -v tabix >/dev/null 2>&1; then
echo "ERROR: tabix not found in PATH" >&2
exit 1
fi
mkdir -p "$(dirname "$OUT_GFF")"
tmp_dir="$(mktemp -d)"
trap 'rm -rf "$tmp_dir"' EXIT
normalized="$tmp_dir/normalized.gff3"
headers="$tmp_dir/headers.gff3"
body="$tmp_dir/body.gff3"
sorted_body="$tmp_dir/body.sorted.gff3"
sorted_all="$tmp_dir/sorted.gff3"
# Read plain or gzip-compressed GFF3.
if [[ "$IN_GFF" == *.gz ]]; then
reader="gzip -cd"
else
reader="cat"
fi
# Normalize Ensembl-style seqids to UCSC-style seqids.
# Examples:
# 1 -> chr1
# 13 -> chr13
# X -> chrX
# Y -> chrY
# MT -> chrM
$reader "$IN_GFF" | awk 'BEGIN{OFS="\t"}
/^#/ {print; next}
NF == 9 {
if ($1 ~ /^[0-9]+$/) $1 = "chr" $1;
else if ($1 == "X") $1 = "chrX";
else if ($1 == "Y") $1 = "chrY";
else if ($1 == "MT") $1 = "chrM";
print;
next;
}
' > "$normalized"
# Preserve headers, then sort records by seqid and start coordinate.
grep '^#' "$normalized" > "$headers" || true
grep -v '^#' "$normalized" > "$body" || true
sort -k1,1 -k4,4n "$body" > "$sorted_body"
cat "$headers" "$sorted_body" > "$sorted_all"
# Compress and index.
bgzip -c "$sorted_all" > "$OUT_GFF"
tabix -f -p gff "$OUT_GFF"
echo "Wrote: $OUT_GFF"
echo "Wrote: $OUT_GFF.tbi"
Make it executable:
chmod +x scripts/prepare_gff3_for_jbrowse.sh
Run it:
scripts/prepare_gff3_for_jbrowse.sh \
raw/Homo_sapiens.GRCh38.115.gff3.gz \
ref/annotations/hg38/Homo_sapiens.GRCh38.115.ucsc.sorted.gff3.gz
Validate the result:
tabix -l ref/annotations/hg38/Homo_sapiens.GRCh38.115.ucsc.sorted.gff3.gz | head
Expected output should use chr names if the FASTA uses chr names:
chr1
chr2
chr3
...
Also check that a known region returns features:
tabix ref/annotations/hg38/Homo_sapiens.GRCh38.115.ucsc.sorted.gff3.gz \
chr13:24882135-24885186 | head
If this command returns gene records, JBrowse should also be able to display the gene annotation track in that region.
Place repeat-mask files under:
ref/masks/<genome_name>/
Recommended hg38 example:
ref/masks/hg38/hg38.no_alt.masking.v2.sorted.gff3.gz
ref/masks/hg38/hg38.no_alt.masking.v2.sorted.gff3.gz.tbi
The repeat-mask GFF3 file must also be:
bgziptabix -p gffIf the repeat-mask file uses Ensembl-style seqids while the FASTA uses chr names, convert it with the same scripts/prepare_gff3_for_jbrowse.sh script:
scripts/prepare_gff3_for_jbrowse.sh \
raw/hg38.no_alt.masking.v2.gff3.gz \
ref/masks/hg38/hg38.no_alt.masking.v2.sorted.gff3.gz
Validate:
tabix -l ref/masks/hg38/hg38.no_alt.masking.v2.sorted.gff3.gz | head
Check FASTA contigs:
cut -f1 ref/genomes/hg38/hg38.no_alt.fa.gz.fai | head
Check gene annotation contigs:
tabix -l ref/annotations/hg38/Homo_sapiens.GRCh38.115.ucsc.sorted.gff3.gz | head
Check repeat-mask contigs:
tabix -l ref/masks/hg38/hg38.no_alt.masking.v2.sorted.gff3.gz | head
The names should match. For a chr-style hg38 FASTA, all three should use chr1, chr2, etc.
Test a JBrowse region manually:
tabix ref/annotations/hg38/Homo_sapiens.GRCh38.115.ucsc.sorted.gff3.gz \
chr13:24882135-24885186 | head
If the command returns records, but JBrowse still shows an empty track, check the generated JBrowse config and confirm it points to the same prepared .gff3.gz file.
A pre-built image is published on Docker Hub:
docker pull aryashta/twb:latest
Available tags: latest, v0.7.3.
This starts the app with empty bundled-resource dropdowns. Users can still upload custom files through the UI.
docker run --rm -p 8000:8000 aryashta/twb:latest
Mount the local ref/ directory into the container:
docker run --rm \
-p 8000:8000 \
-v "$(pwd)/ref:/app/ref:ro" \
aryashta/twb:latest
The mounted ref/ directory is read-only inside the container. This is recommended because bundled genome files should not be modified by the application.
Use:
docker compose up
Recommended runtime mounts:
services:
twb:
image: aryashta/twb:latest
ports:
- "8000:8000"
volumes:
- ./uploads:/app/uploads
- ./output:/app/output
- ./job_status:/app/job_status
- ./jbrowse_configs:/app/jbrowse_configs
- ./ref:/app/ref:ro
The Docker image does not include bundled reference data.
docker build -t aryashta/twb .
Twitcher itself is not compiled during the build. The image downloads
the prebuilt twitcher-linux-x86_64 binary from the pinned Twitcher
release and verifies it against the sha256 checksum the release publishes
alongside it, then installs it to /usr/local/bin/twitcher. This is
faster and more reproducible than compiling Twitcher with Cargo on every
build.
The pinned version lives in the TWITCHER_VERSION build arg (default set
in the Dockerfile, currently v0.7.3). Override it to build against a
different release:
docker build --build-arg TWITCHER_VERSION=v0.7.4 -t aryashta/twb .
docker run --rm aryashta/twb twitcher --version
docker run --rm aryashta/twb python --version
docker run --rm aryashta/twb twitcher-postprocess --help
.fa, .fasta, .fa.gz, .fasta.gz).Visualization controls:
The web UI supports user-created filter presets.
Typical workflow:
Imported presets are added to the preset library. They are not applied automatically. Select an imported preset and apply it explicitly when needed.
The web UI also supports an explicit filter-combination mode:
For multi-value categorical fields, has_any means OR within one cell and has_all means AND within one cell. For example, gene_names has_all BRCA1,TP53 requires the same observation's pipe-separated gene_names field to contain both values.
When reproducing a web filter in the CLI, pass the corresponding -A all or -A any option explicitly.
Preset JSON files use frontend-style filter objects, for example:
{
"schema": "twitcher_filter_presets",
"version": 1,
"exportedAt": "2026-06-03T11:02:00.000Z",
"presets": [
{
"id": "preset_1780492344850_ybn021t5lg",
"name": "Perfect 30bp identity",
"filters": [
{
"col": "identity_30bp_upstream",
"op": "eq",
"val": "1"
},
{
"col": "identity_30bp_downstream",
"op": "eq",
"val": "1"
}
]
}
]
}
Supported filter operators are:
eq, neq, contains, not_contains, starts_with, ends_with,
gt, lt, gte, lte, empty, notempty, in, notin,
has_any, has_all, has_none,
region_overlaps, region_within, region_contains, region_distance_lt
See "Supported filter operators" in docs/CLI.md
for the full list with usage notes, including the region and CIGAR operators
for columns such as cluster_region, ts_1_4_region, and ts_cigar_ctx.
twitcher-postprocess is the command-line entry point for preprocessing, repeated-event grouping, legacy deduplication, labeling, and filtering of Twitcher CSV files outside the web UI.
Full CLI documentation, including every mode, flag, and filter operator, lives in docs/CLI.md. This section only covers the essentials: installation and the entry point.
The current dedup_cli.py supports three main modes:
The CLI entry point is defined in pyproject.toml:
[project.scripts]
twitcher-postprocess = "backend.dedup_cli:main"
git clone <repo>
cd <repo>
pip install -e .
Check the installed CLI:
twitcher-postprocess --help
See docs/CLI.md for: inspecting available columns, full post-processing mode, repeated-event handling, read/sample labeling, group filters, filter-only mode and every supported filter operator (including region and CIGAR operators), advanced query filtering, filter reports, dedup-only mode, efficiency tips, and running the CLI inside Docker.
See the scoring_system.
Content type
Image
Digest
sha256:641d95494…
Size
578 MB
Last updated
1 day ago
docker pull aryashta/twb