Python worker that runs data pipeline code.
3.0K
Github Repo: https://github.com/dataplane-app/dataplane
The best way to get start is to use the quick start docker compose file. Instructions at https://github.com/dataplane-app/dataplane
Environment variables that have a secret_ prefix are loaded as secrets. Dataplane will redact the secret if it is found in any logs.
| Environment variable | Description |
|---|---|
| secret_db_host | Host of the Postgresql database |
| secret_db_user | User for connection to Postgresql database |
| secret_db_pwd | Password for connection to Postgresql database |
| secret_db_ssl | One of disable, allow, prefer, require, verify-ca, verify-full - https://www.postgresql.org/docs/current/libpq-ssl.html |
| secret_db_port | Database port |
| secret_db_database | Database name, default dataplane |
| secret_jwt_secret | Generate a UUID secret for JWT. It is important that you keep this secret safe. To create a secret, you can use an online generator for example https://www.uuidgenerator.net/ |
| secret_encryption_key | Generate a 32 charater long random password. It is important you keep this password safe. You can use an online generator for example https://www.lastpass.com/features/password-generator |
| Environment variable | Options | Example | Description |
|---|---|---|---|
| DP_NATS | nats://nats:4222, nats://nats-r_1:4222 | Connection string to NATS | |
| DP_DEBUG | "true", "false" | false | Print debug logs to console. Recommended to turn off in production. |
| DP_DB_DEBUG | "true", "false" | false | Print database debug logs to console. |
| DP_MQ_DEBUG | "true", "false" | false | Print message queue debug logs to console. |
| DP_METRIC_DEBUG | "true", "false" | false | Print CPU and memory metrics debug logs to console. |
| DP_WORKER_HEARTBEAT_SECONDS | 1 | The interval in seconds that the worker sends a heart beat to the main app. | |
| DP_WORKER_GROUP | python_1 | The worker group is the collection of worker nodes that have the same configuration. For example, a python worker group that runs the python scripts in the pipeline. | |
| DP_WORKER_CMD | /bin/bash | The shell command installed on the linux. This is useful for different linux installations. | |
| DP_WORKER_TYPE | "container", "other" | container | The worker type is for CPU and memory metrics collection. This can differ between a containerised or bare metal installation. If unsure, recommended to keep it to "other". |
| DP_WORKER_LB | "roundrobin" | roundrobin | The load balancer strategy is how analytical workloads are distributed to worker nodes. |
| DP_WORKER_ENV | Development | This is the name of the environment the worker node belongs to. This must match environments set inside the main app. | |
| DP_WORKER_PORT | 9005 | The port that the worker node runs on. |
Content type
Image
Digest
sha256:939564369…
Size
381.9 MB
Last updated
about 2 years ago
docker pull dataplane/dataplane-worker-python