An Integration Agent is a zero-maintenance ETL engine which runs behind the company’s firewall.
4.8K
The Integration Agent runs Etlworks data pipelines inside your own network, behind your firewall. This image includes the Agent, Java runtime, and bundled JDBC drivers.
Install and manage it with standard Docker commands. No native installer, host Java installation, or host Agent CLI is required.
You need an Etlworks account and an Integration Agent license. Start a free trial.
Etlworks connects databases, APIs, files, and queues for ETL, ELT, CDC, and real-time pipelines. Build flows visually or with SQL, JavaScript, and Python, then schedule and monitor them in one workspace. Simba, the built-in AI agent, helps build and diagnose flows. Etlworks runs in the cloud, on-premises, or in hybrid deployments.
The Integration Agent executes Etlworks flows on infrastructure you control. It lets flows reach databases, files, and applications that the Etlworks host cannot access directly.
Flow processing occurs on the Agent. Where data is read, written, or staged depends on your flow configuration. The Agent exchanges configuration, execution status, and logs with Etlworks.
The Integration Agent executes flows. Simba, the built-in AI agent, helps you build and manage them.
Run Simba Prompt flows execute on the Etlworks app. They cannot run on an Integration Agent. A nested flow or Composer flow containing a Run Simba Prompt step must also execute on the app host.
Update the app and Agent together for connector options that require newer runtime capabilities, then wait for the Agent's next status update before assigning those flows.
No inbound port is required for communication between the Agent and Etlworks. Flows still need connectivity to their configured sources and destinations.
| Tag | Purpose |
|---|---|
<version>-java25 | A version-qualified Java 25 image |
latest | Moving tag for the release published as latest |
The image build targets linux/amd64 and linux/arm64. Windows and macOS hosts run the Linux image through Docker Desktop or another compatible Docker environment.
Use a published tag matching your deployment and verify its platform availability. For reproducible deployments, pin the image by digest:
etlworks/integration-agent@sha256:<published-digest>
The Java 25 image includes Azul Zulu OpenJDK. You do not install Java on the host.
.alic license file. License instructions.A starting allocation is 2 CPU cores and 4 GB RAM. Adjust resources for your workloads, especially CDC, large datasets, and parallel flows. On Docker Desktop, also check its resource allocation.
Linux / macOS — Bash or Zsh
Choose a writable host location. The following example uses /opt/etlworks/agent:
mkdir -p /opt/etlworks/agent/{config,license,data,logs,plugin,jdbc,jdbc-custom,backups}
cd /opt/etlworks/agent
Creating directories under /opt may require administrator privileges. On Docker Desktop, ensure the chosen location is available for file sharing.
Windows — Command Prompt
mkdir C:\etlworks\agent
cd /d C:\etlworks\agent
for %D in (config license data logs plugin jdbc jdbc-custom backups) do mkdir "%D"
Inside a .bat file, use %%D instead of %D.
Copy the downloaded .alic file into the host's license directory.
Do not rename the license file. Use the license downloaded for this Agent.
For normal UI-managed operation, the license supplies the Agent identity and Etlworks server address. You do not need to extract a configuration template or manually fill in agent-id, client, or etlworks.url.
Replace REPLACE_WITH_PUBLISHED_TAG with the tag you want to deploy.
Linux / macOS — Bash or Zsh
IMAGE="etlworks/integration-agent:REPLACE_WITH_PUBLISHED_TAG"
docker pull "$IMAGE"
docker run -d --name integration-agent \
--hostname integration-agent \
--init \
--restart unless-stopped \
--stop-timeout 360 \
--memory 4g \
-v /opt/etlworks/agent/config:/opt/etlworks-agent/config \
-v /opt/etlworks/agent/license:/opt/etlworks-agent/license \
-v /opt/etlworks/agent/data:/opt/etlworks-agent/data \
-v /opt/etlworks/agent/logs:/opt/etlworks-agent/logs \
-v /opt/etlworks/agent/plugin:/opt/etlworks-agent/plugin \
-v /opt/etlworks/agent/jdbc:/opt/etlworks-agent/jdbc \
-v /opt/etlworks/agent/jdbc-custom:/opt/etlworks-agent/jdbc-custom \
-v /opt/etlworks/agent/backups:/opt/etlworks-agent/backups \
"$IMAGE"
Windows — Command Prompt
set "IMAGE=etlworks/integration-agent:REPLACE_WITH_PUBLISHED_TAG"
docker pull "%IMAGE%"
docker run -d --name integration-agent ^
--hostname integration-agent ^
--init ^
--restart unless-stopped ^
--stop-timeout 360 ^
--memory 4g ^
-v C:\etlworks\agent\config:/opt/etlworks-agent/config ^
-v C:\etlworks\agent\license:/opt/etlworks-agent/license ^
-v C:\etlworks\agent\data:/opt/etlworks-agent/data ^
-v C:\etlworks\agent\logs:/opt/etlworks-agent/logs ^
-v C:\etlworks\agent\plugin:/opt/etlworks-agent/plugin ^
-v C:\etlworks\agent\jdbc:/opt/etlworks-agent/jdbc ^
-v C:\etlworks\agent\jdbc-custom:/opt/etlworks-agent/jdbc-custom ^
-v C:\etlworks\agent\backups:/opt/etlworks-agent/backups ^
"%IMAGE%"
Replace the host paths if you created the Agent home elsewhere. Container paths remain /opt/etlworks-agent/.... Quote each complete volume argument if its host path contains spaces.
At startup, the image:
etlworks user.These commands work in Bash, Zsh, PowerShell, and Windows Command Prompt:
docker logs --tail 100 integration-agent
docker inspect --format "{{.State.Status}}" integration-agent
docker inspect --format "{{.State.Health.Status}}" integration-agent
Confirm that the Agent appears online in Etlworks, then test a flow.
Docker health checks confirm that the Agent JVM is running. They do not establish successful licensing, connectivity to Etlworks, or successful flow execution.
Compose is optional. Save this as compose.yaml in the host's Agent home and replace the image tag:
services:
integration-agent:
image: etlworks/integration-agent:REPLACE_WITH_PUBLISHED_TAG
container_name: integration-agent
hostname: integration-agent
init: true
restart: unless-stopped
stop_grace_period: 6m
mem_limit: 4g
volumes:
- ./config:/opt/etlworks-agent/config
- ./license:/opt/etlworks-agent/license
- ./data:/opt/etlworks-agent/data
- ./logs:/opt/etlworks-agent/logs
- ./plugin:/opt/etlworks-agent/plugin
- ./jdbc:/opt/etlworks-agent/jdbc
- ./jdbc-custom:/opt/etlworks-agent/jdbc-custom
- ./backups:/opt/etlworks-agent/backups
Start and inspect the deployment:
docker compose pull
docker compose up -d
docker compose logs --tail 100
Choose either the docker run deployment or Compose; do not create both with the same container name.
Agent home inside the container is /opt/etlworks-agent.
| Directory | Purpose |
|---|---|
config | Local configuration. Missing defaults are created automatically; existing files are preserved. |
license | Downloaded Agent license. |
data | Working data and persisted execution state, including CDC state where applicable. |
logs | Agent and flow logs. |
plugin | Additional and premium connector drivers. |
jdbc | Active JDBC driver tree. Bundled driver folders are refreshed at startup. |
jdbc-custom | Customer-managed JDBC overlay applied to the active driver tree. |
backups | Storage available for backups created by your maintenance procedures or applicable tooling. |
The examples persist all these directories. Mounting backups does not automatically create backups during a standard Docker update.
The entrypoint initially runs with root privileges to prepare directories and ownership, then drops privileges to the Agent user, UID 10001 by default. Host storage must allow those operations. Read-only mounts or storage that forbids ownership changes may prevent startup.
Docker's memory limit controls the container's total memory allocation:
--memory 4g
In Compose:
mem_limit: 4g
Without an explicit heap override, the Agent uses:
-XX:MaxRAMPercentage=70.0
Some cloud-storage uploads require a temporary local copy. Keep the JVM temporary directory writable and allow free disk space for the file being uploaded. The staged copy is removed after success or failure.
JAVA_OPTS remains supported. It applies to the Agent JVM, not the setup and health-check JVMs.
For example, add this to docker run:
-e JAVA_OPTS="-Xms100m -Xmx2800m"
Or to the Compose service:
environment:
JAVA_OPTS: "-Xms100m -Xmx2800m"
Leave room within the container limit for native memory, threads, and other JVM overhead. Do not set the heap equal to the entire container memory limit.
Java 25 module-access options and finite XML/XSLT compatibility limits are applied automatically. Explicit JAVA_OPTS values override the defaults; quoted arguments are supported, but shell expansion is not.
The image includes the IPWorks X12/EDIFACT compatibility library; a valid EDI license is still required. If an older image needed a separate workaround, remove its manual -javaagent option and obsolete compatibility mount when upgrading. Preserve other JVM options and mounts.
Image builds for Agent 10.1.0 and later include Git and the OpenSSH client for flows that materialize a Git repository or run dbt from Git. Verify the tools in the image you deploy:
docker exec integration-agent git --version
docker exec integration-agent ssh -V
These commands work from Bash, Zsh and Windows CMD. For Compose, use
docker compose exec <service-name> git --version and
docker compose exec <service-name> ssh -V with your Agent service name.
Configure the repository URL, authentication and SSH known-hosts information in the flow's Git Repository connection. The container needs outbound access to the repository, commonly HTTPS on port 443 or SSH on port 22, or the repository's configured port. No new inbound port is required. Local Git checkout destinations that must survive container replacement belong in a writable persistent data mount.
For an existing container that lacks these tools, use the image replacement procedure under Upgrading, preserving its license, configuration and data mounts. An Agent runtime archive update does not add operating-system packages to a container. Install Git in the image, not on the Docker host; a native Agent installer or host Agent CLI is not required for Docker.
The local configuration file is:
config/integration-agent.properties
It is created automatically on first startup if missing. Edit the host copy when you need local configuration changes, then restart the container:
docker restart integration-agent
For Compose:
docker compose restart
Normal UI-managed Agents also receive configuration from Etlworks. Use the Etlworks UI for settings managed there; received configuration can affect the effective local values.
Do not rely on watch.configuration.changes=true for live reload in UI-managed operation.
Configure connectors, CDC options, and JavaScript compatibility in Etlworks. Update both the app and Agent before enabling options that require a newer Agent capability. Persist CDC offsets/schema history and any explicitly configured metadata directories under writable mounted storage. Do not delete a running task's state.
For an Agent running as an EKS pod, configure EKS Pod Identity or IRSA for its Kubernetes service account and grant the required S3/KMS permissions. In an updated Amazon S3 (SDK) connection, enable Use AWS default credential chain and leave access-key, secret-key, and external-ID fields empty. Keep projected tokens readable by UID/GID 10001. Other default-chain credentials can take precedence; test on the intended Agent. Plain Docker does not create an EKS identity. See AWS workload identity guidance.
The Docker image listens for dashboard requests on 0.0.0.0:8081 (web.server.host / web.server.port). web.server.enabled defaults to true. No host port is published in the examples unless you add it.
shutdown.timeout defaults to 300000 milliseconds. The examples allow six minutes for Docker shutdown. If increasing the Agent timeout, increase Docker's stop timeout too.
Email alerts require valid mail.smtp.* settings and the corresponding notification.* flags. Put Java-properties comments on separate lines.
The quick-start commands do not publish a port. Publishing the dashboard is optional and is not required for communication with Etlworks.
To make it accessible only from the Docker host, add:
-p 127.0.0.1:8081:8081
For Compose:
ports:
- "127.0.0.1:8081:8081"
If you change web.server.port, update the container-side port in the mapping too.
The image includes bundled JDBC drivers. Startup refreshes the bundled driver folders while retaining additional customer folders.
Update the complete image to keep bundled JDBC drivers, authentication libraries, and runtime dependencies together. Startup refreshes bundled JDBC folders, including Hive and Athena. Customer overrides in jdbc-custom and plugin are preserved; review them when upgrading because older JARs can replace updated dependencies or cause authentication failures.
For explicit Athena AwsCredentialsProviderClass settings, check the driver migration guidance when upgrading. Keep SQL Server JDBC and Azure authentication libraries together for Entra authentication.
Place customer-managed drivers in jdbc-custom, preserving the directory structure expected under jdbc.
For example, an overlay file at:
jdbc-custom/<driver-folder>/<driver-file>.jar
is applied to:
jdbc/<driver-folder>/<driver-file>.jar
Use the folder required by the connector. Restart the container after changing drivers.
For Java 25 images, custom drivers must support Java 25, including the authentication method you use. An older driver in jdbc-custom replaces the bundled driver and can reintroduce authentication failures.
Premium connectors need their driver and a separate connector license, in addition to the Agent's .alic license.
The application's Settings → Premium connectors controls do not install drivers on Integration Agents. Install the required driver on each Agent that executes the flow. Do not copy the application's managed-drivers folder to an Agent.
plugin directory.RTK=YOUR_RUNTIME_KEY in the premium connection's Other parameters, then save the connection. This key is used by connections executing on the Agent and does not require machine activation. Keep real keys out of Dockerfiles and command history.Keep the examples' fixed hostname when replacing containers, especially for trials or machine-bound licenses. Persisting license files alone does not preserve machine identity. RTK avoids machine activation.
If Etlworks support has supplied a machine-bound trial/license procedure for your driver, run its activation as the Agent user so license files are created beside the driver on persistent storage:
docker exec -it --user etlworks integration-agent bash
Inside that Linux container shell:
cd /opt/etlworks-agent/plugin
java -jar name.of.the.driver.jar -license
Use the actual driver filename and vendor instructions. Keep resulting license files in the mounted plugin directory. RTK does not replace the Agent's .alic file.
See Managing premium connectors for the application-server workflow and licensing terminology.
With the updated Java 25 app and Agent, OpenEDI and EDIFACT/X12 credentials can be provisioned through the Etlworks license configuration and delivered to the Agent automatically. They are no longer embedded in the image.
To override provisioned credentials on this Agent, set these properties in the
host's config/integration-agent.properties, then restart the container:
edination.api.key=REPLACE_WITH_OPENEDI_API_KEY
ipworks.edi.license=REPLACE_WITH_IPWORKS_RUNTIME_LICENSE
Nonempty local values override the provisioned defaults. OpenEDI still needs network access to its API endpoint. Offline EDIFACT/X12 execution also requires any non-bundled schemas to be available locally.
docker logs --tail 100 integration-agent
docker logs --follow integration-agent
docker stop --time 360 integration-agent
docker start integration-agent
docker restart integration-agent
Compose equivalents:
docker compose logs --tail 100
docker compose logs --follow
docker compose stop
docker compose start
docker compose restart
Docker updates replace the image. There is no native Agent updater service or native bridge migration inside the container.
Stop the old container with:
docker stop --time 360 integration-agent
After backing up the persistent directories, remove it:
docker rm integration-agent
Then repeat the appropriate docker run command with the replacement image.
Record the previous image reference and keep a retrievable copy for recovery. Update the image reference in compose.yaml, then pull it:
docker compose pull
docker compose stop
Back up the persistent directories, then recreate the service:
docker compose up -d
docker compose logs --tail 100
Direct Docker and Compose updates do not provide automatic backup restoration or transactional rollback. Retain the previous image reference and your backup until verification is complete. Do not use docker compose down -v: it removes Docker-managed volumes. To recover from a failed replacement, stop the candidate, restore the matching persistent-folder backup if needed, and recreate the previous image with the original mounts and settings.
Older images used /agent inside the container. The Java 25 image uses /opt/etlworks-agent.
/agent/data to /opt/etlworks-agent/data.config, jdbc, jdbc-custom, and backups directories if they were not previously mounted.license directory.JAVA_OPTS for Java 25 compatibility. Memory options such as -Xms and -Xmx remain supported.Do not run the native 9→10 migration scripts inside the container.
On normal startup, the entrypoint populates missing configuration files automatically.
If startup reports that integration-agent.properties is missing, inspect earlier log messages and check whether the mounted directory allows file creation and ownership changes.
Confirm that the downloaded .alic file is in the host directory mounted at:
/opt/etlworks-agent/license
Do not rename it. Restart after correcting the mount or copying the file.
Check the address embedded in the downloaded license and verify that it is reachable from inside the container, including DNS, port, proxy routing, and TLS trust.
localhost inside the container refers to the container itself.Also verify connectivity from the container to the databases, APIs, and other services used by your flows.
Docker health checks confirm that the Agent JVM is present. Check the Agent logs for licensing, authentication, or connectivity errors, and verify status in the Etlworks UI.
Inspect the logs and Docker container state for an out-of-memory condition. Review both the container memory limit and JAVA_OPTS, leaving sufficient space outside the Java heap.
Content type
Image
Digest
sha256:82f40638e…
Size
1.6 GB
Last updated
about 18 hours ago
docker pull etlworks/integration-agent