flybase/harvdev-gene-identifier repository overview

⁠harvdev-gene-identifier

⁠Introduction.

This repo is a helper script to get the latest data from a FlyBase database and dockerise and run
https://github.com/grivaz/FlyBaseAnnotationHelper⁠ and
https://huggingface.co/cgrivaz/FlyBaseGeneAbstractClassifier⁠

See these for information on how the code all works. NOTE: To generate the data using this repo you will need access to a FlyBase postgres database. If you do not have this then you must generate the data another way but call the file names the same things.

⁠Installation

1. Clone this repository.
2. Build docker image OR pull from dockerhub
   1. docker build . -t gene-identifier
   2. docker pull flybase/harvdev-gene-identifier

NOTE: gene-identifier being used in examples below. Switch names in commands if using the pulled image.

⁠Environment variables needed

1. USER - FlyBase postgres db user name
2. PGPASSWORD - FlyBase postgres password
3. SERVER - FlyBase postgres server if using local db instance you need to use host.docker.internal
4. PORT - FlyBase postgres port
5. DB - FlyBase postgres db name
6. PORT - FlyBase postgres port
7. MONDAY_DATE - start date used to generate new Pubs in FlyBase.
8. GI_DATA_INPUT - local directory to store files needed to run gene-identifier (*optional, can change docker command directly)
9. GI_DATA_OUTPUT - local directory to put output from gene-identifier (*optional)

⁠Data files needed (skip this if generating these in interactive shell)

1. Generate a list of Dmel and Hsap current gene synonyms (fb_synonym_latest.tsv) docker run --rm -p$PORT:$PORT -v $GI_DATA_INPUT:/src/input/ -e SERVER=$SERVER -e PGPASSWORD=$PGPASSWORD -e USER=$USER -e DB=$DB -e PORT=$PORT --entrypoint /usr/bin/python3 gene-identifier src/get_synonyms_batch.py --filepath /src/input/

2. Generate a list of Dmel and Hsap gene unique names (currentDmelHsap.txt) docker run --rm -p$PORT:$PORT -v $GI_DATA_INPUT:/src/input/ -e SERVER=$SERVER -e PGPASSWORD=$PGPASSWORD -e USER=$USER -e DB=$DB -e PORT=$PORT --entrypoint /usr/bin/python3 gene-identifier src/get_gene_uniquenames.py --filepath /src/input/

3. Get PMC ids file (PMC-ids.csv) docker run --rm -v $GI_DATA_INPUT:/src/input/ --entrypoint /usr/bin/bash gene-identifier src/get_PMC.sh

4. Get PMC's to examine (new_pub_dbxrefs.txt) docker run --rm -p$PORT:$PORT -v $GI_DATA_INPUT:/src/input/ -e SERVER=$SERVER -e PGPASSWORD=$PGPASSWORD -e MONDAY_DATE=$MONDAY_DATE -e USER=$USER -e DB=$DB -e PORT=$PORT --entrypoint /usr/bin/python3 gene-identifier src/get_new_pubs.py --filepath /src/input/ Note you can also create this by hand by just adding a list on PMC identifiers.

⁠Running

1. Run the gene identifier code (interactive mode):

   1. docker run --rm -v $GI_DATA_INPUT:/src/input -e SERVER=$SERVER -e PGPASSWORD=$PGPASSWORD -e USER=$USER -e DB=$DB -e PORT=$PORT -v $GI_DATA_OUTPUT:/usr/src/app/output_files -it gene-identifier
   2. If input files not created yet create them
      1. python3 src/get_synonyms_batch.py --filepath /src/input/
      2. python3 src/get_gene_uniquenames.py --filepath /src/input/
      3. python3 src/get_new_pubs.py --filepath /src/input/
      4. sh src/get_PMC.sh
   3. Change to the directory FlyBaseAnnotationHelper by running cd FlyBaseAnnotationHelper
   4. Execute the command python3 update_resources.py
   5. Execute the command python3 annotation_helper.py /usr/src/app/output_files/new_pub_dbxrefs.txt
   6. Output file can be found in the output directory, $GI_DATA_OUTPUT outside of docker and /usr/src/app/output_files inside docker

2. Run code on command line locally (via GoCd etc))

   1. Get the files needed by following Datafiles needed section or via alternative methods.
   2. docker run --rm -v $GI_DATA_INPUT:/src/input -v $GI_DATA_OUTPUT:/src/output --entrypoint /usr/bin/bash gene-identifier src/run_gene_identifier.sh