Public | Automated Build

Last pushed: a year ago
Short Description
directory-form-data
Full Description

directory-form-data

Export Directory registration form data service

Build status

Requirements

Docker >= 1.10
Docker Compose >= 1.8

Local installation

$ git clone https://github.com/uktrade/directory-form-data
$ cd directory-form-data
$ make

Running

Requires all host environment variables to be set.

$ make run

Running for local development

Provides defaults for all env vars but AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

$ make run_debug

Running tests

$ make run_test

Environment variables

Environment variable Default value Description
SQS_REGION_NAME eu-west-1 AWS region name
SQS_FORM_DATA_QUEUE_NAME directory-form-data AWS SQS form data queue name
SQS_INVALID_MESAGES_QUEUE_NAME directory-form-data-invalid AWS SQS invalid messages queue name
SQS_WAIT_TIME 20 (max value) AWS SQS Long Polling - how long to wait for messages on a single boto API call
SQS_MAX_NUMBER_OF_MESSAGES 10 (max value) How many messages to receive on a single boto API call
SQS_VISIBILITY_TIMEOUT 21600 (6 hours, max value is 43200) Time after which retrieved but not deleted messages will return to the queue
SECRET_KEY test when running make test and docker-compose locally, otherwise None Django secret key
DATABASE_URL postgres://test:test@localhost:5432/directory-form-data-test, postgres://test:test@postgres:5432/directory-form-data-test for docker-compose Postgres database url
AWS_ACCESS_KEY_ID None, set in .env for local docker-compose AWS access key ID
AWS_SECRET_ACCESS_KEY None, set in .env for local docker-compose AWS secret access key

Host environment variables for docker-compose

.env files will be automatically created (with env_writer.py based on env.json and env-postgres.json) by make run_test, based on host environment variables with DIRECTORY_FORM_DATA_ prefix.

Web server and queue worker

Host environment variable Docker environment variable
DIRECTORY_FORM_DATA_SQS_REGION_NAME SQS_REGION_NAME
DIRECTORY_FORM_DATA_SQS_FORM_DATA_QUEUE_NAME SQS_FORM_DATA_QUEUE_NAME
DIRECTORY_FORM_DATA_SQS_INVALID_MESAGES_QUEUE_NAME SQS_INVALID_MESAGES_QUEUE_NAME
DIRECTORY_FORM_DATA_SQS_WAIT_TIME SQS_WAIT_TIME
DIRECTORY_FORM_DATA_SQS_MAX_NUMBER_OF_MESSAGES SQS_MAX_NUMBER_OF_MESSAGES
DIRECTORY_FORM_DATA_SQS_VISIBILITY_TIMEOUT SQS_VISIBILITY_TIMEOUT
DIRECTORY_FORM_DATA_SECRET_KEY SECRET_KEY
DIRECTORY_FORM_DATA_DATABASE_URL DATABASE_URL
DIRECTORY_FORM_DATA_AWS_ACCESS_KEY_ID AWS_ACCESS_KEY_ID
DIRECTORY_FORM_DATA_AWS_SECRET_ACCESS_KEY AWS_SECRET_ACCESS_KEY

Database

Host environment variable Docker environment variable
DIRECTORY_FORM_DATA_POSTGRES_USER POSTGRES_USER
DIRECTORY_FORM_DATA_POSTGRES_PASSWORD POSTGRES_PASSWORD
DIRECTORY_FORM_DATA_POSTGRES_DB POSTGRES_DB

Architecture

Web server -> Amazon SQS Queue -> Queue worker -> Database

Web server and Queue worker use same Docker image with different CMD, see docker-compose.yml.

Web server

  1. Web server is started with gunicorn.
  2. Receives POST request from directory-form-ui.
  3. Request goes to form view.
  4. If form data is valid, it is sent to Amazon SQS queue $SQS_FORM_DATA_QUEUE_NAME.

Queue worker

  1. Queue worker is started with django management command.
  2. Retrieves messages (the maximum of $SQS_MAX_NUMBER_OF_MESSAGES) from Amazon $SQS_FORM_DATA_QUEUE_NAME.
    1. If there are no messages, it long polls for $SQS_WAIT_TIME.
      1. It keeps making polling calls every $SQS_WAIT_TIME until messages are received.
    2. If messages were retrieved:
      1. If message body is valid form data, a new instance of form.models.Form is saved to the database.
        1. If message with same message ID was already saved (as SQS provides at-least-once delivery), it is skipped (unique constraint on form.models.Form.sqs_message_id).
      2. Else it is send to $SQS_INVALID_MESAGES_QUEUE_NAME.
  3. If SIGTERM or SIGINT signal is received:
    1. If it happened at the start of long polling:
      1. If docker stop wait time (default is 10s) is less than $SQS_WAIT_TIME (default is 20s), docker will kill -9 the worker process (which is OK).
    2. Else if it happened before making another polling call or during processing messages, it will exit gracefully.
      1. Processing of the current message will finish.
      2. Retrieved, but not deleted messages will reappear in the queue after $SQS_VISIBILITY_TIMEOUT
Docker Pull Command
Owner
ukti

Comments (0)