mgaltd/data_snake repository overview

Note: Here is a shortened image description for the ** ETL data_snake ** application, full documentation is available at: doc.etldatasnake.com⁠. Please, also visit the Product page: etldatasnake.com⁠

⁠Quick reference:

• The application is provided and developed by MGA ltd. Poland / Toruń / Piaskowa 6⁠
• Product page etldatasnake.com⁠
• Documentation doc.etldatasnake.com⁠
• Contact us: [email protected]⁠

⁠Supported tags:

Note: Please remember that we always recommend that you use the highest possible version of our software.

Please go to the TAGS⁠ section

⁠What is ETL data_snake:

ETL data_snake is an ETL class software (Extract, Transform, Load). It can be used to migrate data from one source to a different one in an efficient manner. Collect, migrate and transform data from many different sources and save it where you want.

We support cooperation with:

• PostgreSQL
• Oracle
• MySQL
• Microsoft SQL Server
• Fusion Registry >= v10 (SDMX Statistical Data Warehouse)
• MS Excel files
• CSV files
• Other sources and targets using customs solutions via. python scripts.

⁠How to use this image

The following services must be started for the application to run properly:

• Main application container - command: gunicorn
• Main Celery asynchronous job queue - command: celery
• Asynchronous job queue responsible for process execution - command: processes_celery
• System Load Monitor - command: system_monitor_celery
• The container responsible for establishing the application states (migrations for the database and a collection of static files) - command: set_state
• Application web server mgaltd/data_snake_webserver⁠
• Application Database PostgreSQL⁠
• Redis⁠

⁠docker-compose

The most convenient way to start ETL data_snake is to use the example docker-compose.yml file below

Note: After the first launch of the application, you can log in with a predefined administrator account ** Login **: admin ** Password **: admin

Note: The first launch of the application may take up to several minutes.

Note: To function properly, the application requires uploading a license key

Note: By default, the web server will deliver the application on ports 80 and 443. By default, the application is delivered by a web server with a self-signed SSL certificate. It is possible to add your own certificate (see Volumes⁠ section)

version: "3.5"

services:

    redis:
        image: redis:6.0.5-alpine

    postgres:
        image: postgres:12-alpine
        restart: always
        environment: &pgenv
            POSTGRES_DB: "data_snake"
            POSTGRES_USER: "data_snake"
            POSTGRES_PASSWORD: "data_snake"
        volumes:
            - data_snake_postgresql:/var/lib/postgresql/data

    # Web application for ETL data_snake using gunicorn
    etl-app: &etl-app
        image: mgaltd/data_snake:latest
        restart: always
        depends_on:
            - postgres
            - redis
        volumes:
            - data_snake_media/:/opt/etl/media
            - data_snake_static/:/opt/etl/static
            - data_snake_logs/:/opt/etl/logs/
            - data_snake_files/:/opt/etl/files/
        hostname: example.domain.com
        environment:
            <<: *pgenv
            POSTGRES_HOST: "postgres"
            POSTGRES_PORT: 5432
            REDIS_HOST: "redis"
            REDIS_PORT: 6379
            REDIS_DB: 0
            ETL_SECRET_KEY: "__this__must__be__change__on__production__server__"
            ETL_DEBUG: "False"  #  !!! This must be string
            # This is the address used for sharing and process api token;
            # when using proxy this is the public address (i.e. mga.com.pl)
            ETL_EXTERNAL_URL: "example.domain.com"

        command: gunicorn

    # Service runs celery
    service-celery: &celery
        <<: *etl-app
        command: celery

    # Service runs processes celery
    service-processes-celery:
        <<: *celery
        command: processes_celery

    # Service runs system monitor celery
    service-system-monitor-celery:
        <<: *celery
        command: system_monitor_celery

    # That service starts only once, does collectstatic, migrate and dies
    etl-set-state:
        <<: *etl-app
        restart: "no"
        depends_on:
            - etl-app
        command: set_state

    etl-nginx:
        image: mgaltd/data_snake_webserver:latest
        depends_on:
            - etl-app
            - etl-set-state
        volumes:
            - data_snake_media/:/opt/etl/media
            - data_snake_static/:/opt/etl/static
            # - data_snake_ssl/certificate.pem:/opdata_snake/ssl/certificate.pem
            # - data_snake_ssl/key.pem:/opt/etl/ssl/key.pem
        ports:
            - 80:80
            - 443:443

volumes:
    data_snake_media:
    data_snake_static:
    data_snake_logs:
    data_snake_files:
    data_snake_ssl:
    data_snake_postgresql:

⁠Volumes

In order for the application to function properly it is essential to mount resources. The application stores the following resources:

• data_snake_media - contains a set of static files
• data_snake_static - static files such as css, js, photos, etc. Static files are shared with the web server via the volume.
• data_snake_logs - space for application logs
• data_snake_files - shared space for sources/targets files. All files as sources or targets should be stored in this directory.
• data_snake_ssl - possibility to mount the directory containing the SSL certificate. By default, the application runs with a self-signed SSL certificate. If you need to add your own certificate, mount the following files:

  
  # In etl-nginx service
  volumes:
  
  data_snake_ssl/certificate.pem:/opdata_snake/ssl/certificate.pem
  data_snake_ssl/key.pem:/opt/etl/ssl/key.pem
  

  Copy

⁠Environmental Variables

Note: Here is a short list of the most important environmental variables that you may want to set up at the very beginning. For a complete list, see the application's documentation Configuration⁠

• ETL_SECRET_KEY - A secret key for a particular ETL data_snake software installation. This is used to provide cryptographic signing of data, and should be set to a unique, unpredictable value. Refer to external documentation for more information. Default: "__this__must__be__change__on__production__server__"

• ETL_AUDIT_IS_ACTIVE - ETL data_snake has a builtin auditing mechanism that tracks changes in application data. This mechanism generates a large ammount of additional data and may slow down bulk operations. It can be disabled by changing this variable to "False" (as a string). Default: "True"

• ETL_TIME_ZONE - The timezone used by the ETL data_snake software. This is used for process scheduling. Default: "Europe/Warsaw"

• GUNICORN_TIMEOUT - The amount of time after which worker processes that are not responding will be restarted. Worker processes are responsible for handling requests and returning a response to the client. Value is in seconds. Default: 300

• ETL_SENTRY_DSN - The DSN to use to connect to Sentry. Default: unset

• POSTGRES_HOST - Name of the host for the PostgreSQL database used by the ETL data_snake software to store application data. Value is an internal docker address of the PostgreSQL container. Default: "postgres"

• POSTGRES_PORT - The TCP port for the PostgreSQL database used by the ETL data_snake software to store application data. Default: 5432

• REDIS_DB - Which of the Redis databases to use for connecting to the Redis cache. Default: 0

• REDIS_HOST -Name of the host for the Redis cache used by the ETL data_snake software. Value is an internal docker address of the Redis container. Default: "redis"

• REDIS_PORT - The TCP port for the Redis cache used by the ETL data_snake software. Default: 6379