watchlog/agent repository overview

⁠Watchlog Agent Docker Image

The Watchlog Agent Docker image provides a quick, consistent way to deploy the Watchlog Agent for collecting and sending logs and metrics to Watchlog⁠. This guide covers pulling the image, configuring the agent, and running it with the proper runtime options to monitor system and container metrics.

⁠Key Features

• Streamlined deployment via Docker.
• Collect host-level metrics (CPU, memory, disk, network, logs, and service metrics).
• Support for multiple database integrations (PostgreSQL, MySQL, Redis, MongoDB).
• Optional container-level isolation.
• Secure transport over HTTPS.
• Custom log parsing with auto or regex patterns.
• Easy configuration via integration.json file.

⁠Prerequisites

• Docker Engine (17.06 or later) installed on your system.
• Watchlog API key and server URL (provided in your Watchlog panel).

⁠Configuration

The Watchlog Agent uses file-based configuration via integration.json. All integrations are configured in this single file, making it easy to manage multiple services and instances.

⁠File-based Configuration

The easiest way to configure the agent is by mounting an integration.json file. This allows you to easily manage multiple integrations and update configurations without modifying Docker commands.

⁠Step 1: Create integration.json

Create a file named integration.json with your integrations:

[
  {
    "service": "postgresql",
    "monitor": false,
    "host": "127.0.0.1",
    "port": "5432",
    "username": "postgres",
    "password": "your_password",
    "database": ["mydb"]
  },
  {
    "service": "postgresql",
    "monitor": false,
    "host": "127.0.0.1",
    "port": "5433",
    "username": "postgres",
    "password": "your_password",
    "database": ["anotherdb"]
  },
  {
    "service": "mysql",
    "monitor": false,
    "host": "127.0.0.1",
    "port": "3306",
    "username": "root",
    "password": "your_password",
    "database": ["mydb"]
  },
  {
    "service": "redis",
    "monitor": false,
    "host": "127.0.0.1",
    "port": "6379",
    "password": ""
  },
  {
    "service": "mongodb",
    "monitor": false,
    "host": "127.0.0.1",
    "port": "27017",
    "username": "",
    "password": ""
  },
  {
    "service": "docker",
    "monitor": false
  },
  {
    "service": "gitlab",
    "monitor": false,
    "baseUrl": "https://gitlab.com",
    "token": "your_token",
    "projects": ["project-id-1", "project-id-2"],
    "monitorLogs": true,
    "metrics": {
      "pipelines": true,
      "jobs": true,
      "runners": true,
      "mrs": true,
      "deployments": true
    }
  },
  {
    "service": "nginx",
    "monitor": true,
    "accessLog": "/var/log/nginx/access.log"
  }
]

⁠Step 2: Run with mounted configuration

docker run -d \
  --name watchlog-agent \
  --network host \
  -v /path/to/your/integration.json:/app/app/config/integration.json \
  -e WATCHLOG_APIKEY="your_api_key" \
  -e WATCHLOG_SERVER="https://your.watchlog.server" \
  watchlog/agent:latest

⁠Or using Docker volume:

# Create volume
docker volume create watchlog-agent-config
⁠Copy your integration.json to the volume (or create it directly)
docker run --rm -v watchlog-agent-config:/data -v /path/to/your/integration.json:/source/integration.json alpine cp /source/integration.json /data/integration.json
⁠Run container
docker run -d 

--name watchlog-agent 

--network host 

-v watchlog-agent-config:/app/app/config 

-e WATCHLOG_APIKEY="your_api_key" 

-e WATCHLOG_SERVER="https://your.watchlog.server⁠" 

watchlog/agent:latest

⁠Required Environment Variables

Only two environment variables are required:

• WATCHLOG_APIKEY: Your Watchlog API key. (required)
• WATCHLOG_SERVER: URL of your Watchlog server (e.g. https://log.watchlog.io). (required)
• UUID (optional): Override the default host identifier.

Note: All integrations are configured in the integration.json file. You do not need to set any additional environment variables for integrations.

⁠Log Monitoring (Watchlist)

The Watchlog Agent can monitor log files from other containers using Docker volumes. Create a log-watchlist.json file in the same directory as your integration.json:

⁠Step 1: Create log-watchlist.json

[
  {
    "name": "app-logs",
    "path": "/shared-logs/app.log",
    "service": "application",
    "format": "auto"
  },
  {
    "name": "nginx-access",
    "path": "/shared-logs/nginx/access.log",
    "service": "nginx",
    "format": "auto"
  }
]

⁠Step 2: Mount Shared Volumes

To access logs from other containers, you need to mount a shared volume. Here are two common approaches:

⁠Method 1: Shared Docker Volume (Recommended)

Create a shared volume and mount it to both containers:

# Create a shared volume for logs
docker volume create shared-logs
⁠Run your application container with the shared volume
docker run -d 

--name my-app 

-v shared-logs:/app/logs 

my-app-image
⁠Run watchlog-agent with the same shared volume
docker run -d 

--name watchlog-agent 

--network host 

-v /path/to/integration.json:/app/app/config/integration.json 

-v /path/to/log-watchlist.json:/app/app/config/log-watchlist.json 

-v shared-logs:/shared-logs 

-e WATCHLOG_APIKEY="your_api_key" 

-e WATCHLOG_SERVER="https://your.watchlog.server⁠" 

watchlog/agent:latest

In your log-watchlist.json, use paths relative to the mounted volume (e.g., /shared-logs/app.log).

⁠Method 2: Bind Mount from Host

If your application writes logs to the host filesystem, you can bind mount them:

# Run your application container (logs written to host)
docker run -d \
  --name my-app \
  -v /var/log/my-app:/app/logs \
  my-app-image
⁠Run watchlog-agent with access to host logs
docker run -d 

--name watchlog-agent 

--network host 

-v /path/to/integration.json:/app/app/config/integration.json 

-v /path/to/log-watchlist.json:/app/app/config/log-watchlist.json 

-v /var/log/my-app:/host-logs 

-e WATCHLOG_APIKEY="your_api_key" 

-e WATCHLOG_SERVER="https://your.watchlog.server⁠" 

watchlog/agent:latest

In your log-watchlist.json, use paths relative to the mounted directory (e.g., /host-logs/app.log).

⁠Log Format Options

• "format": "auto" - Automatically detect log format (nginx, pm2, redis, mysql, docker, postgresql, mongodb)
• "format": "custom" - Use a custom regex pattern:

  {
    "name": "custom-logs",
    "path": "/shared-logs/custom.log",
    "service": "custom",
    "format": "custom",
    "pattern": "^(?<date>\\d{4}-\\d{2}-\\d{2}) \\[(?<level>\\w+)\\] (?<message>.*)$"
  }

  Copy

⁠Backward Compatibility

You can still use the environment variable LOG_WATCHLIST_JSON if the file is not found:

-e LOG_WATCHLIST_JSON='[{"name":"app-logs","path":"/shared-logs/app.log","service":"application","format":"auto"}]'

⁠Multiple Integrations Support

The Watchlog Agent supports monitoring multiple instances of the same service type. Simply add multiple entries with the same service name in your integration.json file.

⁠Example: Multiple PostgreSQL Databases

Monitor multiple PostgreSQL databases on different ports:

[
  {
    "service": "postgresql",
    "monitor": true,
    "host": "127.0.0.1",
    "port": "5432",
    "username": "postgres",
    "password": "pass",
    "database": ["db1", "db2"]
  },
  {
    "service": "postgresql",
    "monitor": true,
    "host": "127.0.0.1",
    "port": "5433",
    "username": "postgres",
    "password": "pass",
    "database": ["db3"]
  }
]

⁠Example: Multiple Redis Instances

[
  {
    "service": "redis",
    "monitor": true,
    "host": "127.0.0.1",
    "port": "6379",
    "password": ""
  },
  {
    "service": "redis",
    "monitor": true,
    "host": "127.0.0.1",
    "port": "6380",
    "password": "redis_password"
  }
]

⁠Example: Multiple MySQL Databases

[
  {
    "service": "mysql",
    "monitor": true,
    "host": "127.0.0.1",
    "port": "3306",
    "username": "root",
    "password": "pass",
    "database": ["db1"]
  },
  {
    "service": "mysql",
    "monitor": true,
    "host": "127.0.0.1",
    "port": "3307",
    "username": "root",
    "password": "pass",
    "database": ["db2", "db3"]
  }
]

Important: Each integration is automatically identified by a unique ID based on host:port combination. Make sure each instance has a unique host/port combination.

⁠Basic Usage Example

1. Pull the image:

   docker pull watchlog/agent:latest

   Copy
2. Run the container:

   docker run -d \
     --name watchlog-agent \
     --network host \
     -v /path/to/integration.json:/app/app/config/integration.json \
     -e WATCHLOG_APIKEY="your_api_key" \
     -e WATCHLOG_SERVER="https://your.watchlog.server" \
     watchlog/agent:latest

   Copy
3. Verify startup:

   docker logs -f watchlog-agent

   Copy

⁠Network Configuration

When running in Docker, you may need to use --network host to access services on the host machine, or use host.docker.internal as the host address in your configuration.

For Mac/Windows: Use host.docker.internal as the host in your integration.json

For Linux: Use --network host and 127.0.0.1 or localhost as the host

⁠Support

If you encounter any issues, please check the official docs⁠ or reach out via [email protected]⁠.