stackateam/fdj_worker repository overview

⁠FACTURES FDJ WORKER

Runs on EC2-docker AWS instance

⁠FACUTRE process description:

1. the rabbitMQ consummed is "facture_queue_prod""
2. The worker consumes this element from the queue :

{
  "docid": O,

  "algo": "fdj",
}

1. Get the s3_key and bucket for pdf download
2. Process of this pdf for data extraction
3. Push of the produced data (data+analysis_processed) towards POSTGRES DB
4. Acknowledgment sent to rabbit so that it can delete the element from the queue

⁠How to use me (Docker image):

⁠Build :

docker build -t stackateam/fdj_worker:0.5 . --rm

⁠Run :

Locally

docker run --env-file env.list stackateam/fdj_worker:<tag_version>

On EC2 instance

nohup sudo docker run --env-file env.list stackateam/fdj_worker:latest > caseip_logs.txt &

⁠env.list Format:

The file "env.list" should contain the following variables:

AWS_SECRET_ACCESS_KEY=
AWS_ACCESS_KEY_ID=

RABBIT_QUEUE=
RABBIT_USER=
RABBIT_PASSWORD=
AMPQ_URL=

HEROKU_API_TOKEN=
HEROKU_API=

⁠How to run the code (not from docker image):

pipenv run python worker.py