stoix/dataflow-runner repository overview

⁠Dataflow Runner in Docker

Starts a Dataflow Flex Template job on Google Dataflow and waits for the execution to finish returning with exit code 0 on success and a non-zero value for failure.

Maintained by STOIX⁠.

⁠How to use this image

See Template Dataflow Python⁠ for an example.

Example of a local run using Docker:

docker run -it \
  -e ... \ # Add required environment variables
  stoix/dataflow-runner:LATEST

⁠Environment Variables

The following environment variables are used to configure the Dataflow Runner. The required variables need to be provided, the rest have default values listed.

⁠GCP_PROJECT_ID

Google Cloud project id for running the Dataflow job.

Default: The Dataflow Runner tries to infer the project when authenticating.

⁠GCP_REGION (required)

Google Cloud region⁠ for the Dataflow job worker machines. E.g: europe-north1.

⁠GCP_SERVICE_ACCOUNT

Base64 encoded string of service account json to run the Dataflow job.

Default: If not set, the Dataflow Runner falls back on Google Cloud authentication flow⁠ for trying to infer authentication.

E.g:

{
  "type": "service_account",
  "project_id": "stoix",
  "private_key_id": "...",
  "private_key": "...",
  "client_email": "[email protected]",
  "client_id": "...",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "..."
}

Produces the following Base64 encoded string:

ewogICJ0eXBlIjogInNlcnZpY2VfYWNjb3VudCIsCiAgInByb2plY3RfaWQiOiAic3RvaXgiLAogICJwcml2YXRlX2tleV9pZCI6ICIuLi4iLAogICJwcml2YXRlX2tleSI6ICIuLi4iLAogICJjbGllbnRfZW1haWwiOiAiZGF0YWZsb3dAc3RvaXguaWFtLmdzZXJ2aWNlYWNjb3VudC5jb20iLAogICJjbGllbnRfaWQiOiAiLi4uIiwKICAiYXV0aF91cmkiOiAiaHR0cHM6Ly9hY2NvdW50cy5nb29nbGUuY29tL28vb2F1dGgyL2F1dGgiLAogICJ0b2tlbl91cmkiOiAiaHR0cHM6Ly9vYXV0aDIuZ29vZ2xlYXBpcy5jb20vdG9rZW4iLAogICJhdXRoX3Byb3ZpZGVyX3g1MDlfY2VydF91cmwiOiAiaHR0cHM6Ly93d3cuZ29vZ2xlYXBpcy5jb20vb2F1dGgyL3YxL2NlcnRzIiwKICAiY2xpZW50X3g1MDlfY2VydF91cmwiOiAiLi4uIgp9

⁠JOB_IMAGE (required)

The Flex Template image to run. See Template Dataflow Python⁠ for an example.

⁠JOB_NAME_PREFIX

Prefix used to build the Dataflow job name. Full name will be <prefix>-YYYY-MM-DD-HH-mm-ss.

Default: stoix-dataflow-runner

⁠JOB_SDK_LANGUAGE

Beam SDK⁠ language used for the Dataflow Flex Template. Possible values: JAVA or PYTHON.

Default: Tries to read java or python from the image name.

⁠JOB_PARAM_<NAME>

Used to add pipeline arguments. The <NAME> part will be extracted, lowercased and provided to the pipeline.

E.g: Given JOB_PARAM_INPUT_FILE=gs://bucket/file.txt the Dataflow Runner will provide --input_file=gs://bucket/file.txt to the Dataflow Flex Template job.

⁠MACHINE_TYPE

The Compute Engine machine type that Dataflow uses when starting worker VMs. You can use any of the available Compute Engine machine type families as well as custom machine types.

The Dataflow service chooses the machine type based on your job if you do not set this option.

⁠MAX_WORKERS

The maximum number of Compute Engine instances to be made available to your pipeline during execution. Note that this can be higher than the initial number of workers (specified by num_workers) to allow your job to scale up, automatically or otherwise.

If unspecified, the Dataflow service determines an appropriate number of workers.

⁠NUM_WORKERS

The number of Compute Engine instances to use when executing your pipeline.

If unspecified, the Dataflow service determines an appropriate number of workers.

⁠DISK_SIZE_GB

The disk size, in gigabytes, to use on each remote Compute Engine worker instance. If set, specify at least 30 GB to account for the worker boot image and local logs.

Omit or set to 0 to use the default size defined in your Cloud Platform project.

⁠Use with STOIX

When used on STOIX⁠, the following Dataflow Flex Template parameters are automatically populated:

⁠--stoix_scheduled

Set to the RFC3339 date time for the job schedule.

E.g: 2021-01-01T00:00:00Z