vestalisvirginis/synphage repository overview

⁠synphage

Pipeline to create phage genome synteny graphics from genbank files

This library provides an intuitive tool to create synteny graphics highlighting the conserved genes between multiple genome sequences.
This tool is primarily designed to work with phage genomes or other short sequences of interest, although it works with bacterial genomes as well.

Despite numerous synteny tools available on the market, this tool has been conceived because none of the available tools allows to visualise gene conservation in multiple sequences at one glance (as typically cross-links are drawn only between two consecutive sequences for a better readability).

As a result synphage was born.

In addition to show conserved genes across multiple sequences, the originality of this library stands in the fact that when working on the same set of genomes the initial blast and computation need to be run only once. Multiple graphics can then be generated from these data, comparing all the genomes or only a set of genomes from the analysed dataset. Moreover, the generated data is also available to the user as a table, where individual genes or groups of genes can easily be checked by name for conservation or uniqueness in the other genomes.

⁠Stats

⁠Install

docker pull vestalisvirginis/synphage:<tag>

Replace with the desired version.

⁠Usage

⁠Setup

'synphage' setup is as follow:

• /user_files : folder that will receive the users genbank files
• /data: folder that contains all the data generated during the run

synphage requires the user to specify the following environment variables:

• EMAIL (optional): to connect to the NCBI database.
• API_KEY (optional): to connect to the NCBI database and download files.

⁠Running Synphage

A step-by-step example, performed on a group of closely related Lactococcus phages is available in the documentation⁠.

⁠Starting Dagster

synphage uses Dagster⁠. In order to run synphage jobs, you need to start dagster first.

Set up the environment variable DAGSTER_HOME in order to keep a trace of your previous run (optional). For more information, see Dagster documentation⁠.

docker run -p 3000 vestalisvirginis/synphage:<tag>

For more information and options, check running synphage container⁠.

⁠Running the jobs

synphage pipeline is composed of four steps that need to be run sequencially. See complete documention⁠

⁠Step 1: Loading the data into the pipeline

Data is loaded into the pipeline from the input_folder set by the user and/or downloaded from the NCBI.

• step_1a_get_user_data : load user's data
• step_1b_download : download data from the NCBI

⁠Query config options :

⁠Step 2: Data validation

Completeness of the data is validated at this step.

• step_2_make_validation : perform checks and transformations on the dataset that are required for downstream processing

⁠Step 3: Blasting the data

The blast is performed at this step of the pipeline and three different options are available:

• step_3a_make_blastn : run a Nucleotide BLAST on the dataset
• step_3b_make_blastp : run a Protein BLAST on the dataset
• step_3c_make_all_blast : run both, Nucleotide and Protein BLAST simultaneously

⁠Step 4: Synteny plot

The graph is created during this last step. The step 4 can be run multiple times with different configurations and different sets of data, as long as the data have been processed once through steps 1, 2 and 3.

• step_4_make_plot : use data generated at step 3 and the genbank files to plot the synteny diagram

⁠Plotting config options

The appearance of the plot can be modified through the configuration.

⁠Output

synphage's output consists of four to six main parquet files (depending if blastn and blastp were both executed) and the synteny graphic. However all the data generated by the synphage pipeline are made available in your data directory.

⁠Generated data architecture

.
├── <path_to_synphage_folder>/
│   ├── download/
│   ├── fs/
│   ├── genbank/
│   ├── gene_identity/
│   │   ├── fasta_n/
│   │   ├── blastn_database/
│   │   └── blastn/
│   ├── protein_identity/
│   │   ├── fasta_p/
│   │   ├── blastp_database/
│   │   └── blastp/
│   ├── tables/
│   │   ├── genbank_db.parquet
│   │   ├── processed_genbank_df.parquet
│   │   ├── blastn_summary.parquet
│   │   ├── blastp_summary.parquet
│   │   ├── gene_uniqueness.parquet
│   │   └── protein_uniqueness.parquet
│   ├── sequences.csv
│   └── synteny/
│      ├── colour_table.parquet
│      ├── synteny_graph.png
│      └── synteny_graph.svg
└── ...

⁠Tables

The tables folder contains the four to six main parquet files generated by the pipeline.

1. genbank_db.parquet : original data parsed from the GenBank files.
2. processed_genbank_df.parquet : data processed during the validation step. It contains two additional columns:
   • gb_type : specifying what type of data is used as unique identifier of the coding elements
   • key: unique identifier based on the columns: filename, id and locus_tag.
3. blastn_summary.parquet : data parsed from the blastn output json files. It contains the collection of the best match for each sequence against each genomes. The percentage of identity between two sequences are then used for calculating the plot cross-links between the sequences.
4. blastp_summary.parquet : data parsed from the blastp output json files. It contains the collection of the best match for each sequence against each genomes. The percentage of identity between two sequences are then used for calculating the plot cross-links between the sequences.
5. gene_uniqueness.parquet : combines both processed_genbank_df.parquet and blastn_summary.parquet in a single parquet file, allowing the user to quickly know how many matches their sequence(s) of interest has/have retrieved. These data are then used to compute the colour code used for the synteny plot. The result of the computation is recorded in the colour_table.parquet. This file is over-written between each plot run.
6. protein_uniqueness.parquet : combines both processed_genbank_df.parquet and blastp_summary.parquet in a single parquet file, allowing the user to quickly know how many matches their sequence(s) of interest has/have retrieved. These data are then used to compute the colour code used for the synteny plot. The result of the computation is recorded in the colour_table.parquet. This file is over-written between each plot run.

⁠Synteny plot

The synteny plot is generated as .svg file and .png file, and contains the sequences indicated in the sequences.csv file. The genes are colour-coded according to their abundance (percentage) among the plotted sequences. The cross-links between each consecutive sequence indicates the percentage of similarities between those two sequences.

⁠Documentation

Visit https://vestalisvirginis.github.io/synphage/⁠ for complete installation instruction⁠, guidelines to navigate the pipeline⁠ and step-by-step example⁠.

⁠Support

Where to ask for help?

Open a discussion⁠.

⁠Roadmap

• [x] create config options for the plot at run time
• [x] integrate the NCBI search
• [x] implement blastp
• [] create possibility to add ref sequence with special colour coding
• [] create interactive plot
• [] Help us in a discussion?

⁠Status

[2024-07-20] ✨ New features!

• Checks : to validate the quality of the data

• Blastp is finally implemented

[2024-01-11] ✨ New feature! to simplify the addition of new sequences into the genbank folder

• download : download genomes to be analysed from the NCBI database

⁠Contributing

We accept different types of contributions, including some that don't require you to write a single line of code. For detailed instructions on how to get started with our project, see CONTRIBUTING⁠ file.

⁠Authors

• vestalisvirginis⁠ / Virginie Grosboillot / 🇫🇷

⁠License

Apache License 2.0 Free for commercial use, modification, distribution, patent use, private use. Just preserve the copyright and license.