sanderdms/w0_ingest repository overview

⁠w0_ingest Docker Image

The w0_ingest Docker image is designed to simplify the operation of the w0_ingest node application offered by wardr0be⁠, which loads Dusk blockchain data into your PostgreSQL database. The image supports two operational modes: RUES (live block and transaction ingestion) and Batch mode (syncing from the last block in the database). You can run multiple containers with different configurations to suit your needs.

⁠Two Main Operation Modes:

1. RUES Mode (-e NPM_COMMAND=rues): Consumes RUES websockets to live ingest blocks and transactions as they occur.
2. Batch Mode (-e NPM_COMMAND=batch): Syncs data starting from the last block in the database and catches up to the latest block.

The image also includes an integrated cron feature for scheduling batch operations at regular intervals.

⁠Example Uses:

⁠1. RUES Container (Default)

Run a default configured container that captures transactions as they hit the mempool and ingest blocks as soon as they become available.

docker run -e POSTGRES_URL=postgresql://user:[email protected]:5432 sanderdms/w0_ingest

This is equivalent to:

docker run \
  -e POSTGRES_URL=postgresql://postgres:[email protected]:5432 \
  -e META_PROCESS_NAME=RUES_INGEST_DEFAULT_DOCKER \
  -e RUSK_NODE_ADDRESS=https://testnet.nodes.dusk.network \
  -e RUSK_NODE_PORT=443 \
  -e LOG_LEVEL=debug \
  -e ENABLE_CRON=false \
  -e NPM_COMMAND=rues \
  sanderdms/w0_ingest

⁠2. Batch Container

Run a batch container to perform a full load and catch up on any missed events from RUES mode. The container below will run an _ingest job every minute to stay up-to-date.

docker run \
  -e POSTGRES_URL=postgresql://postgres:[email protected]:5432 \
  -e META_PROCESS_NAME=BATCH_INGEST_DEFAULT_DOCKER \
  -e ENABLE_CRON=true \
  -e TASK_SCHEDULE='*/1 * * * *'
  -e NPM_COMMAND=batch \
  sanderdms/w0_ingest

⁠Advanced Usage:

⁠Example JSON Logging:

To enable JSON logging, use the following configuration. This will create a log file inside the container and mount a volume to your host machine for easy access.

docker run \
  -e POSTGRES_URL=postgresql://postgres:[email protected]:5432 \
  -e JSON_LOGFILE_PATH="log/example_log.json" \
  -e LOG_LEVEL=debug \
  -v ./log:/usr/src/app/_ingest/log \
  sanderdms/w0_ingest

⁠Log Levels:

Possible log levels include: silly, info, warn, debug, error.

⁠Note on Database Hostnames:

If you are running PostgreSQL inside a Docker container, use host.docker.internal as the hostname in the connection URL.

⁠Note on Networking:

There is no DNS caching build in to the image. Adviced to have a stable DNS on your host machine. If in doubt, configure --dns 1.1.1.1 to your container to use Cloudflare DNS service to resolve the RUSK_NODE_ADDRESS. There is retry mechanism in place, but if a new block rolls in before previous was processed it can cause issues with blocktime calculation and lead to 0 blocktime. If you see EAI_AGAIN in your logs you have a DNS issue.