etlworks/integration-agent repository overview

⁠Etlworks Integration Agent

The Integration Agent runs Etlworks data pipelines inside your own network, behind your firewall. This image includes the Agent, Java runtime, and bundled JDBC drivers.

Install and manage it with standard Docker commands. No native installer, host Java installation, or host Agent CLI is required.

You need an Etlworks account and an Integration Agent license. Start a free trial⁠.

⁠What is Etlworks?

Etlworks⁠ is an any-to-any data integration platform for ETL, ELT, change data capture (CDC), EDI, APIs, files, queues, and IoT. Connect your systems, transform data, and orchestrate pipelines from a single workspace.

Build pipelines on a drag-and-drop canvas, write transformations in SQL, JavaScript, or Python, and automate operations through the CLI and APIs. Simba, the built-in AI agent⁠, helps you build flows, diagnose errors, and work with your data using natural language.

Capabilities include log-based CDC, real-time streaming, schema drift handling, data validation, dbt integration, and pipelines as code. Schedule pipelines and monitor execution, performance, and audit history in the same application.

Etlworks runs in the cloud, on-premises, or in a hybrid deployment with Integration Agents executing flows inside your network.

⁠What is the Integration Agent?

The Integration Agent executes Etlworks flows on infrastructure you control. It lets flows reach databases, files, and applications that the Etlworks host cannot access directly.

Flow processing occurs on the Agent. Where data is read, written, or staged depends on your flow configuration. The Agent exchanges configuration, execution status, and logs with Etlworks.

⁠How it works

1. Create an Agent in the Etlworks UI and download its license.
2. Start this image with the license and persistent storage mounted.
3. The Agent initiates outbound communication with Etlworks to receive configuration and work.
4. Manage flow execution, schedules, and monitoring from the Etlworks UI.

No inbound port is required for communication between the Agent and Etlworks. Flows still need connectivity to their configured sources and destinations.

⁠Tags and supported platforms

The image build targets linux/amd64 and linux/arm64. Windows and macOS hosts run the Linux image through Docker Desktop or another compatible Docker environment.

Use a published tag matching your deployment and verify its platform availability. For reproducible deployments, pin the image by digest:

etlworks/integration-agent@sha256:<published-digest>

The Java 25 image includes Azul Zulu OpenJDK. You do not install Java on the host.

⁠Prerequisites

• An Etlworks account on a cloud or on-premise instance.
• An Agent created in the Etlworks UI.
• The downloaded Agent .alic license file. License instructions⁠.
• Docker Engine, or Docker Desktop using Linux containers.
• Outbound connectivity from the container to the Etlworks address embedded in the license and to the sources and destinations used by your flows.

A starting allocation is 2 CPU cores and 4 GB RAM. Adjust resources for your workloads, especially CDC, large datasets, and parallel flows. On Docker Desktop, also check its resource allocation.

⁠Quick start

⁠1. Create persistent directories

Linux / macOS — Bash or Zsh

Choose a writable host location. The following example uses /opt/etlworks/agent:

mkdir -p /opt/etlworks/agent/{config,license,data,logs,plugin,jdbc,jdbc-custom,backups}
cd /opt/etlworks/agent

Creating directories under /opt may require administrator privileges. On Docker Desktop, ensure the chosen location is available for file sharing.

Windows — Command Prompt

mkdir C:\etlworks\agent
cd /d C:\etlworks\agent
for %D in (config license data logs plugin jdbc jdbc-custom backups) do mkdir "%D"

Inside a .bat file, use %%D instead of %D.

⁠2. Install the license

Copy the downloaded .alic file into the host's license directory.

Do not rename the license file. Use the license downloaded for this Agent.

For normal UI-managed operation, the license supplies the Agent identity and Etlworks server address. You do not need to extract a configuration template or manually fill in agent-id, client, or etlworks.url.

⁠3. Pull and run the image

Replace REPLACE_WITH_PUBLISHED_TAG with the tag you want to deploy.

Linux / macOS — Bash or Zsh

IMAGE="etlworks/integration-agent:REPLACE_WITH_PUBLISHED_TAG"

docker pull "$IMAGE"

docker run -d --name integration-agent \
  --init \
  --restart unless-stopped \
  --stop-timeout 360 \
  --memory 4g \
  -v /opt/etlworks/agent/config:/opt/etlworks-agent/config \
  -v /opt/etlworks/agent/license:/opt/etlworks-agent/license \
  -v /opt/etlworks/agent/data:/opt/etlworks-agent/data \
  -v /opt/etlworks/agent/logs:/opt/etlworks-agent/logs \
  -v /opt/etlworks/agent/plugin:/opt/etlworks-agent/plugin \
  -v /opt/etlworks/agent/jdbc:/opt/etlworks-agent/jdbc \
  -v /opt/etlworks/agent/jdbc-custom:/opt/etlworks-agent/jdbc-custom \
  -v /opt/etlworks/agent/backups:/opt/etlworks-agent/backups \
  "$IMAGE"

Windows — Command Prompt

set "IMAGE=etlworks/integration-agent:REPLACE_WITH_PUBLISHED_TAG"

docker pull "%IMAGE%"

docker run -d --name integration-agent ^
  --init ^
  --restart unless-stopped ^
  --stop-timeout 360 ^
  --memory 4g ^
  -v C:\etlworks\agent\config:/opt/etlworks-agent/config ^
  -v C:\etlworks\agent\license:/opt/etlworks-agent/license ^
  -v C:\etlworks\agent\data:/opt/etlworks-agent/data ^
  -v C:\etlworks\agent\logs:/opt/etlworks-agent/logs ^
  -v C:\etlworks\agent\plugin:/opt/etlworks-agent/plugin ^
  -v C:\etlworks\agent\jdbc:/opt/etlworks-agent/jdbc ^
  -v C:\etlworks\agent\jdbc-custom:/opt/etlworks-agent/jdbc-custom ^
  -v C:\etlworks\agent\backups:/opt/etlworks-agent/backups ^
  "%IMAGE%"

Replace the host paths if you created the Agent home elsewhere. Container paths remain /opt/etlworks-agent/.... Quote each complete volume argument if its host path contains spaces.

At startup, the image:

• Creates missing configuration files from bundled defaults.
• Preserves existing configuration files.
• Prepares directory ownership, then runs the Agent as the etlworks user.
• Refreshes bundled JDBC drivers and applies custom driver overlays.

⁠4. Verify startup

These commands work in Bash, Zsh, PowerShell, and Windows Command Prompt:

docker logs --tail 100 integration-agent
docker inspect --format "{{.State.Status}}" integration-agent
docker inspect --format "{{.State.Health.Status}}" integration-agent

Confirm that the Agent appears online in Etlworks, then test a flow.

Docker health checks confirm that the Agent JVM is running. They do not establish successful licensing, connectivity to Etlworks, or successful flow execution.

⁠Docker Compose

Compose is optional. Save this as compose.yaml in the host's Agent home and replace the image tag:

services:
  integration-agent:
    image: etlworks/integration-agent:REPLACE_WITH_PUBLISHED_TAG
    container_name: integration-agent
    init: true
    restart: unless-stopped
    stop_grace_period: 6m
    mem_limit: 4g
    volumes:
      - ./config:/opt/etlworks-agent/config
      - ./license:/opt/etlworks-agent/license
      - ./data:/opt/etlworks-agent/data
      - ./logs:/opt/etlworks-agent/logs
      - ./plugin:/opt/etlworks-agent/plugin
      - ./jdbc:/opt/etlworks-agent/jdbc
      - ./jdbc-custom:/opt/etlworks-agent/jdbc-custom
      - ./backups:/opt/etlworks-agent/backups

Start and inspect the deployment:

docker compose pull
docker compose up -d
docker compose logs --tail 100

Choose either the docker run deployment or Compose; do not create both with the same container name.

⁠Persistent storage

Agent home inside the container is /opt/etlworks-agent.

The examples persist all these directories. Mounting backups does not automatically create backups during a standard Docker update.

The entrypoint initially runs with root privileges to prepare directories and ownership, then drops privileges to the Agent user, UID 10001 by default. Host storage must allow those operations. Read-only mounts or storage that forbids ownership changes may prevent startup.

⁠Memory and JVM options

Docker's memory limit controls the container's total memory allocation:

--memory 4g

In Compose:

mem_limit: 4g

Without an explicit heap override, the Agent uses:

-XX:MaxRAMPercentage=70.0

⁠Using JAVA_OPTS

JAVA_OPTS remains supported. It applies to the Agent JVM, not the setup and health-check JVMs.

For example, add this to docker run:

-e JAVA_OPTS="-Xms100m -Xmx2800m"

Or to the Compose service:

environment:
  JAVA_OPTS: "-Xms100m -Xmx2800m"

Leave room within the container limit for native memory, threads, and other JVM overhead. Do not set the heap equal to the entire container memory limit.

Required Java 25 module-access options are applied automatically. JAVA_OPTS supports quoted arguments but does not perform shell expansion.

⁠Configuration

The local configuration file is:

config/integration-agent.properties

It is created automatically on first startup if missing. Edit the host copy when you need local configuration changes, then restart the container:

docker restart integration-agent

For Compose:

docker compose restart

Normal UI-managed Agents also receive configuration from Etlworks. Use the Etlworks UI for settings managed there; received configuration can affect the effective local values.

Do not rely on watch.configuration.changes=true for live reload in UI-managed operation.

⁠Selected template defaults

Email notifications require valid mail.smtp.* settings and the appropriate notification.* flags.

In Java properties files, place comments on separate lines:

# Local dashboard port
web.server.port=8081

⁠Optional dashboard access

The quick-start commands do not publish a port. Publishing the dashboard is optional and is not required for communication with Etlworks.

To make it accessible only from the Docker host, add:

-p 127.0.0.1:8081:8081

For Compose:

ports:
  - "127.0.0.1:8081:8081"

If you change web.server.port, update the container-side port in the mapping too.

⁠JDBC drivers and premium connectors

The image includes bundled JDBC drivers. Startup refreshes the bundled driver folders while retaining additional customer folders.

⁠Custom JDBC drivers

Place customer-managed drivers in jdbc-custom, preserving the directory structure expected under jdbc.

For example, an overlay file at:

jdbc-custom/<driver-folder>/<driver-file>.jar

is applied to:

jdbc/<driver-folder>/<driver-file>.jar

Use the folder required by the connector. Restart the container after changing drivers.

⁠Premium connectors

Premium connectors⁠ require their driver and a separate license.

Copy the driver into the host's plugin directory. If the driver requires interactive activation, open a shell as the Agent user:

docker exec -it --user etlworks integration-agent bash

Then run:

cd /opt/etlworks-agent/plugin
java -jar name.of.the.driver.jar -license

Replace the filename with the actual driver filename. Follow the driver vendor's instructions for license activation and persistence.

Restart the container after changing plugin files.

⁠EDI credentials

With the updated Java 25 app and Agent, OpenEDI and EDIFACT/X12 credentials can be provisioned through the Etlworks license configuration and delivered to the Agent automatically. They are no longer embedded in the image.

To override provisioned credentials on this Agent, set these properties in the host's config/integration-agent.properties, then restart the container:

edination.api.key=REPLACE_WITH_OPENEDI_API_KEY
ipworks.edi.license=REPLACE_WITH_IPWORKS_RUNTIME_LICENSE

Nonempty local values override the provisioned defaults. OpenEDI still needs network access to its API endpoint. Offline EDIFACT/X12 execution also requires any non-bundled schemas to be available locally.

⁠Routine management

docker logs --tail 100 integration-agent
docker logs --follow integration-agent
docker stop --time 360 integration-agent
docker start integration-agent
docker restart integration-agent

Compose equivalents:

docker compose logs --tail 100
docker compose logs --follow
docker compose stop
docker compose start
docker compose restart

⁠Upgrading

Docker updates replace the image. There is no native Agent updater service or native bridge migration inside the container.

⁠Using docker run

1. Record the current image reference and container settings.
2. Pull the replacement image before stopping the Agent.
3. Stop the Agent gracefully.
4. Back up the persistent host directories while it is stopped.
5. Remove the old container.
6. Recreate it using the new image and the same mounts, environment variables, resource limits, and port mappings.
7. Verify logs, Docker health, Agent status in Etlworks, and a test flow.

Stop the old container with:

docker stop --time 360 integration-agent

After backing up the persistent directories, remove it:

docker rm integration-agent

Then repeat the appropriate docker run command with the replacement image.

⁠Using Compose

Update the image reference in compose.yaml, then pull it:

docker compose pull
docker compose stop

Back up the persistent directories, then recreate the service:

docker compose up -d
docker compose logs --tail 100

Direct Docker and Compose updates do not provide automatic backup restoration or transactional rollback. Retain the previous image reference and your backup until verification is complete.

⁠Moving from a pre-10 image

Older images used /agent inside the container. The Java 25 image uses /opt/etlworks-agent.

• Preserve and back up the existing host directories.
• Change the container-side mount paths, such as /agent/data to /opt/etlworks-agent/data.
• Add persistent config, jdbc, jdbc-custom, and backups directories if they were not previously mounted.
• Keep the downloaded Agent license in the mounted license directory.
• Missing configuration defaults are created automatically.
• Review existing JAVA_OPTS for Java 25 compatibility. Memory options such as -Xms and -Xmx remain supported.
• Replace the container with the Java 25 image and verify operation.

Do not run the native 9→10 migration scripts inside the container.

⁠Troubleshooting

⁠The configuration directory is empty

On normal startup, the entrypoint populates missing configuration files automatically.

If startup reports that integration-agent.properties is missing, inspect earlier log messages and check whether the mounted directory allows file creation and ownership changes.

⁠Online license not found

Confirm that the downloaded .alic file is in the host directory mounted at:

/opt/etlworks-agent/license

Do not rename it. Restart after correcting the mount or copying the file.

⁠The Agent cannot connect to Etlworks

Check the address embedded in the downloaded license and verify that it is reachable from inside the container, including DNS, port, proxy routing, and TLS trust.

• localhost inside the container refers to the container itself.
• Cloud deployments normally use HTTPS/443; on-premise deployments may use another port.
• If the Etlworks address changed, obtain the appropriate updated license.
• Custom host mappings are needed only when the container cannot resolve or route to the configured hostname normally.

Also verify connectivity from the container to the databases, APIs, and other services used by your flows.

⁠The container is healthy, but the Agent is offline in Etlworks

Docker health checks confirm that the Agent JVM is present. Check the Agent logs for licensing, authentication, or connectivity errors, and verify status in the Etlworks UI.

⁠The container exits or is killed during a large flow

Inspect the logs and Docker container state for an out-of-memory condition. Review both the container memory limit and JAVA_OPTS, leaving sufficient space outside the Java heap.

⁠Documentation and support

• Using the Integration Agent⁠
• About the Integration Agent⁠
• Install, update, configure and monitor⁠
• Run Etlworks on-premise⁠
• Support and SLAs⁠