etlworks/integration-agent repository overview

⁠What is Etlworks?

Etlworks⁠ is a scalable, any-to-any data integration platform. It supports all modern data integration techniques: ETL, ELT, reverse ETL, CDC, real-time data streaming, API integration, and more.

Choose from hundreds of connectors, formats, and transformations to build complex data integration flows in minutes, no coding required. Specify when and how often you need your flows to run. Monitor running flows in real time.

⁠What is Integration Agent?

NOTE: You must have an active cloud or on-premise Etlworks account in order to use Integration Agent.

An Integration Agent⁠ is a zero-maintenance, easy-to-configure, fully autonomous ETL engine which runs as a background service behind the company’s firewall. It can be installed on Windows and Linux as a standalone app or service and on any OS which supports Docker.

It allows you to run Etlworks Flows that can access on-premise applications and databases. Outbound communication between Integration Agent and Etlworks cloud instance is fully secured and data is never staged in the cloud.

⁠How it works

A few things you should know regarding the Integration Agent:

1. It uses the same ETL engine as Etlworks, so it can run the same Flows as Etlworks, only locally, without going through Etlworks cloud instance.
2. It is fully integrated with the cloud Etlworks instance, so the Flows created in Etlworks Web UI can be triggered to run manually or scheduled to periodically run in the remote Integration Agent.
3. The communication session between the Agent and Etlworks is always initiated by the Agent as we assume that Etlworks cannot access the Agent which is running behind the firewall.
4. When the user triggers any action, such as Run Flow, Stop Flow, Stop Agent the action is added to the queue. The actions in a queue are executed in order on the next communication session between the Agent and the Etlworks Etlworks.

⁠Documentation

Integration Agent documentation can be found here⁠.

⁠Prerequisites

⁠Etlworks account

You must have an active cloud or on-premise Etlworks account. Sign up⁠ for a free Etlworks trial. Other options:

• Install standalone Etlworks app on-premise⁠
• Run Etlworks as a Docker container⁠

⁠Hardware Requirements

Minimum hardware requirements are 2 CPU cores and 4GB RAM. If you are using Docker Desktop make sure that you have enough resources allocated under Resources Settings.

⁠How to use this image

Pull the image from Docker Hub.

docker pull etlworks/integration-agent

⁠Installing license

Step 1. Create root folder for Integration Agent in the host operating system, for example:

cd /opt/etlworks/integration-agent

Step 2. Create license folder under the root folder. For example:

cd /opt/etlworks/integration-agent/license

Step 3. Follow this instruction⁠ to download the license. Copy the license file to the license folder created in step 2.

NOTE: Do not rename license file.

⁠Create and run container with data persistence

Integration agent stores logs in the /logs folder and stages the data if needed in /data folder. To retain data and logs between restarts use the following command to run the container:

Linux/MacOS

docker run -d --name integrationagent \
  -v /path/to/agent/home/logs:/agent/logs \
  -v /path/to/agent/home/data:/agent/data \
  -v /path/to/agent/home/license:/agent/license \
  -v /path/to/agent/home/plugin:/agent/plugin \
  -e JAVA_OPTS="-Xms100m -Xmx4000m" \
  etlworks/integration-agent:latest

Windows

docker run -d --name integrationagent ^
  -v /path/to/agent/home/logs:/agent/logs ^
  -v /path/to/agent/home/data:/agent/data ^
  -v /path/to/agent/home/license:/agent/license ^
  -v /path/to/agent/home/plugin:/agent/plugin ^
  -e JAVA_OPTS="-Xms100m -Xmx4000m" ^
  etlworks/integration-agent:latest

Replace/path/to/agent/home with actual path on the host machine filesystem where you created a root folder for Agent's data and logs.

Modify JAVA_OPTS if needed.

⁠Environment Variables

This image uses optional environment variable JAVA_OPTS. This environment variable can be used to override amount of RAM the container can use in megabytes. The default value is -Xms100m -Xmx4000m

⁠Premium Connectors Drivers

Premium connectors⁠ require special drivers and license.

To add premium connector driver to the container simply copy it to the /path/to/agent/home/plugin directory on the host machine filesystem (create it if needed). Driver license has to be activated within the container:

1. Login to the integration-agent container terminal with docker exec -it integrationagent bash, or from the Docker Desktop UI.
2. Navigate to plugin directory with cd /agent/plugin.
3. Initiate interactive driver license activation with java -jar name.of.the.driver.jar -license, where name.of.the.driver.jar has to be replaced with the actual driver filename.

⁠Upgrades

To upgrade your Integration Agent to the latest version:

Step 1. Stop and remove the old container using terminal (or from the Docker Desktop UI).

docker stop integrationagent
docker rm integrationagent

Step 2. Pull the latest image

docker pull etlworks/integration-agent

Step 3. Start the container

Linux/MacOS

docker run -d --name integrationagent \
  -v /path/to/agent/home/logs:/agent/logs \
  -v /path/to/agent/home/data:/agent/data \
  -v /path/to/agent/home/license:/agent/license \
  -v /path/to/agent/home/plugin:/agent/plugin \
  -e JAVA_OPTS="-Xms100m -Xmx4000m" \
  etlworks/integration-agent:latest

Windows

docker run -d --name integrationagent ^
  -v /path/to/agent/home/logs:/agent/logs ^
  -v /path/to/agent/home/data:/agent/data ^
  -v /path/to/agent/home/license:/agent/license ^
  -v /path/to/agent/home/plugin:/agent/plugin ^
  -e JAVA_OPTS="-Xms100m -Xmx4000m" ^
  etlworks/integration-agent:latest

NOTE: Make sure /path/to/agent/home points to correct directory in the host machine running Integration Agent.

Step 4. Install licenses for premium connectors if needed.