Sign inSign up

stackateam/kbis_worker

By stackateam

•Updated over 6 years ago

Image
0

137

stackateam/kbis_worker repository overview

alt text

⁠KBIS WORKER

Runs on EC2-docker AWS instance

⁠KBIS process description:

  1. the rabbitMQ consummed is "kbis_queue"
  2. The worker consumes this element from the queue :
{
  "docid": O,

  "algo": "kbis",
}
  1. Download of the DLF with GET request from postgres
  2. CASEIP processing of this DLF :
        extractor = Kbisprocessor()
        pred = extractor.preannot(dlf)
  1. Push of the produced data (section+entity+data+analysis_processed) towards POSTGRES DB
  2. Acknowledgment sent to rabbit so that it can delete the element from the queue

⁠How to use me (Docker image):

⁠Build :
docker build -t stackateam/kbis_worker:0.5 . --rm
⁠Run :

Locally

docker run --env-file env.list stackateam/kbis_worker:<tag_version>

On EC2 instance

nohup sudo docker run --env-file env.list stackateam/kbis_worker:latest > caseip_logs.txt &
⁠env.list Format:

The file "env.list" should contain the following variables:

AWS_SECRET_ACCESS_KEY=
AWS_ACCESS_KEY_ID=$

RABBIT_QUEUE=
RABBIT_USER=
RABBIT_PASSWORD=
AMPQ_URL=

HEROKU_API_TOKEN=
HEROKU_API=

⁠How to run the code (not from docker image):

________________________________________________________________
|                           MAN PAGE                           |
| - To run without models DL: python worker.py --no-dwd        |
| - To download all models from S3: python worker.py           |
| - Don't forget to export all the env variables for local use |
|______________________________________________________________|

alt text

Tag summary

Content type

Image

Digest

Size

3.1 GB

Last updated

over 6 years ago

docker pull stackateam/kbis_worker