Sign inSign up

vturb/clickhouse-db-materializer

By vturb

•Updated about 2 months ago

Incrementally materialize MySQL/Postgres/Mongo/DynamoDB into ClickHouse from a single YAML file

Image
0

10K+

vturb/clickhouse-db-materializer repository overview

⁠Running with Docker

The materializer ships as a small (~30 MB) multi-arch container image (linux/amd64 and linux/arm64), published to both Docker Hub and the GitHub Container Registry on every push to master and every v* tag.

RegistryImage
Docker Hubvturb/clickhouse-db-materializer
GHCRghcr.io/vturb/clickhouse-db-materializer

Tags available: latest (tip of master), the semver tags X.Y.Z, X.Y, X for releases, the branch name, and sha-<short> for an exact commit.

docker pull vturb/clickhouse-db-materializer:latest
docker run --rm vturb/clickhouse-db-materializer:latest version

The image's entrypoint is the materializer binary, so anything you pass after the image name is a subcommand and its flags:

docker run --rm vturb/clickhouse-db-materializer:latest --help
docker run --rm vturb/clickhouse-db-materializer:latest functions

⁠Providing a config

The config is a file, so mount it into the container (read-only) and point --config at the mounted path. Secrets stay in the environment via ${VAR} expansion — pass them with -e / --env-file, never bake them into the image.

docker run --rm \
  -v "$PWD/conversions.yaml:/etc/materializer/config.yaml:ro" \
  -e PG_HOST -e PG_USER -e PG_PASSWORD \
  -e CLICKHOUSE_ADDR -e CLICKHOUSE_USER -e CLICKHOUSE_PASSWORD \
  vturb/clickhouse-db-materializer:latest \
  run --config /etc/materializer/config.yaml

Validate a config or do a dry run the same way:

# Validate only
docker run --rm -v "$PWD/conversions.yaml:/etc/materializer/config.yaml:ro" \
  vturb/clickhouse-db-materializer:latest \
  validate --config /etc/materializer/config.yaml

# Transform but never write
docker run --rm -v "$PWD/conversions.yaml:/etc/materializer/config.yaml:ro" \
  --env-file .env \
  vturb/clickhouse-db-materializer:latest \
  run --config /etc/materializer/config.yaml --dry-run

⁠Continuous (daemon) mode

Use --interval to keep syncing on a schedule, and --log-format json for machine-readable logs:

docker run -d --name materializer --restart unless-stopped \
  -v "$PWD/conversions.yaml:/etc/materializer/config.yaml:ro" \
  --env-file .env \
  vturb/clickhouse-db-materializer:latest \
  run --config /etc/materializer/config.yaml --interval 1m --log-format json

The process handles SIGINT/SIGTERM, so docker stop shuts it down cleanly between passes.

⁠docker-compose

services:
  materializer:
    image: vturb/clickhouse-db-materializer:latest
    restart: unless-stopped
    command: ["run", "--config", "/etc/materializer/config.yaml", "--interval", "1m", "--log-format", "json"]
    volumes:
      - ./conversions.yaml:/etc/materializer/config.yaml:ro
    env_file: .env

⁠File-backed state store

The default clickhouse state store is stateless and needs no volume. If you use the file store, mount a writable directory. The container runs as the non-root user nonroot (uid 65532), so the host directory must be writable by that uid:

docker run --rm \
  -v "$PWD/conversions.yaml:/etc/materializer/config.yaml:ro" \
  -v "$PWD/state:/var/lib/materializer" \
  --env-file .env \
  vturb/clickhouse-db-materializer:latest \
  run --config /etc/materializer/config.yaml

with state: { store: file, path: /var/lib/materializer/conversions.json } in the config.

⁠Kubernetes

For continuous sync, a Deployment with one replica works well; for a periodic backfill, a CronJob. Mount the config from a ConfigMap and the secrets from a Secret:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: conversions-materializer
spec:
  replicas: 1
  selector:
    matchLabels: { app: conversions-materializer }
  template:
    metadata:
      labels: { app: conversions-materializer }
    spec:
      containers:
        - name: materializer
          image: vturb/clickhouse-db-materializer:latest
          args:
            - run
            - --config=/etc/materializer/config.yaml
            - --interval=1m
            - --log-format=json
          envFrom:
            - secretRef: { name: conversions-secrets }
          volumeMounts:
            - name: config
              mountPath: /etc/materializer
              readOnly: true
      volumes:
        - name: config
          configMap: { name: conversions-config }

⁠Building the image yourself

docker build \
  --build-arg VERSION="$(git describe --tags --always)" \
  --build-arg COMMIT="$(git rev-parse --short HEAD)" \
  --build-arg DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  -t clickhouse-db-materializer:dev .

⁠How publishing works (maintainers)

The docker-publish.yml⁠ workflow logs in to both registries, computes tags/labels with docker/metadata-action⁠ and pushes a multi-arch image with docker/build-push-action.

  • Docker Hub auth uses the org secrets DOCKERHUB_USERNAME and DOCKERHUB_TOKEN (an access token with write scope).
  • GHCR auth uses the built-in GITHUB_TOKEN (the workflow grants it packages: write).
⁠One-time setup for a brand-new public image

The repository does not exist on Docker Hub yet; the first successful workflow run creates it. Docker Hub creates new repositories as private by default, so after the first push make it public once:

  1. Docker Hub → vturb/clickhouse-db-materializer → Settings → Make public.

Likewise, the GHCR package is created on first push and inherits private visibility. Make it public once at https://github.com/orgs/vturb/packages → the package → Package settings → Change visibility → Public (and, optionally, link it to this repository).

After that one-time step every subsequent push publishes publicly with no further action.

To cut a release, push a tag:

git tag v0.1.0
git push origin v0.1.0

Tag summary

Content type

Image

Digest

sha256:b154f7ca8…

Size

7.6 MB

Last updated

about 2 months ago

docker pull vturb/clickhouse-db-materializer