devjfreaks/nginx-logs-enricher

By devjfreaks

•Updated 8 months ago

Enrich Nginx access logs with geolocation, network, and security data using ipgeolocation.io

Image

Developer tools

Web servers

Monitoring & observability

388

Overview Tags

devjfreaks/nginx-logs-enricher repository overview

⁠nginx-logs-enricher

nginx-logs-enricher is a production-safe, high-performance command-line tool that transforms raw Nginx access logs into actionable intelligence by enriching IP addresses with comprehensive geolocation, network, and security data from ipgeolocation.io. It streams log files line-by-line to handle gigabyte-sized files without memory issues, uses intelligent caching to minimize API calls and costs, employs parallel workers for blazing-fast processing, and gives users complete control over API usage through interactive prompts or automation-friendly flags. Whether you're a DevOps engineer investigating traffic patterns, a security analyst hunting for VPNs and bots, or a SaaS operator analyzing user geography, this tool enriches each unique IP with detailed location data (country, city, coordinates), network information (ASN, ISP, organization), security signals (proxy/VPN/TOR/bot detection, threat scores), abuse contacts, and more—all while maintaining bounded memory usage, respecting API rate limits, and delivering structured JSON output ready for analysis, visualization, or integration into your security workflows.

⁠Key Features

Production-Safe: Streams logs line-by-line, bounded memory, works on gigabyte-sized files
Intelligent Caching: 24-hour cache reduces API calls by 90%+, configurable disk usage limits
Parallel Processing: Concurrent workers with rate limiting for maximum throughput
User Control: Interactive prompts or automation-friendly flags for API usage control
Rich Intelligence: Location, network, security, abuse data from ipgeolocation.io
Docker Ready: Pre-built image, no Go installation needed
Structured Output: JSON Lines format for easy analysis and integration

⁠Quick Start with Docker

⁠Prerequisites

Docker installed on your system
An ipgeolocation.io API key (Get one free⁠)
Nginx access log file

⁠Step 1: Prepare your log file

Make sure your Nginx access log exists in the current directory:

ls -lh access.log

⁠Step 2: Run the enrichment

docker run --rm -it \
  -e IPGEOLOCATION_API_KEY="YOUR_API_KEY" \
  -v "$PWD:/data" \
  devjfreaks/nginx-logs-enricher:latest \
  enrich \
  --input /data/access.log \
  --output /data/enriched.jsonl \
  --db /data/cache.db \
  --workers 10 \
  --rps 5

What this does:

-v "$PWD:/data" → Mounts your current directory into the container at /data
--workers 10 → Uses 10 parallel workers for faster processing
--rps 5 → Limits to 5 API requests per second (prevents rate limiting)

⁠Step 3: View your results

# Check the output files
ls -lh enriched.jsonl cache.db
 
# View enriched data
cat enriched.jsonl | jq '.'
 
# Count traffic by country
cat enriched.jsonl | jq -r '.data.location.country_name' | sort | uniq -c | sort -rn

⁠Common Docker Usage Patterns

⁠Basic enrichment (interactive mode)

docker run --rm -it \
  -e IPGEOLOCATION_API_KEY="YOUR_API_KEY" \
  -v "$PWD:/data" \
  devjfreaks/nginx-logs-enricher:latest \
  enrich --input /data/access.log

⁠Non-interactive mode (for automation/CI)

docker run --rm \
  -e IPGEOLOCATION_API_KEY="YOUR_API_KEY" \
  -v "$PWD:/data" \
  devjfreaks/nginx-logs-enricher:latest \
  enrich \
  --input /data/access.log \
  --enrich-all \
  --no-prompt

⁠High-performance configuration

docker run --rm -it \
  -e IPGEOLOCATION_API_KEY="YOUR_API_KEY" \
  -v "$PWD:/data" \
  devjfreaks/nginx-logs-enricher:latest \
  enrich \
  --input /data/access.log \
  --workers 20 \
  --rps 15 \
  --cache-max-mb 500

⁠Include security signals

docker run --rm -it \
  -e IPGEOLOCATION_API_KEY="YOUR_API_KEY" \
  -v "$PWD:/data" \
  devjfreaks/nginx-logs-enricher:latest \
  enrich \
  --input /data/access.log \
  --include security,abuse

⁠Common Usage Examples

⁠Basic enrichment (default behavior)

./nginx-logs-enricher enrich --input access.log

⁠Include security signals

./nginx-logs-enricher enrich --input access.log --include "security"

⁠Include multiple modules

./nginx-logs-enricher enrich --input access.log --include "security,abuse"

⁠Return only selected fields (smaller output)

./nginx-logs-enricher enrich \
  --input access.log \
  --fields "location.country_name,location.city,asn.organization"

⁠Exclude fields you don't need

./nginx-logs-enricher enrich \
  --input access.log \
  --excludes "currency,location.country_flag"

⁠High-performance enrichment with parallel workers

./nginx-logs-enricher enrich \
  --input access.log \
  --workers 20 \
  --rps 10

⁠Advanced cache configuration

./nginx-logs-enricher enrich \
  --input access.log \
  --cache-ttl-hours 24 \
  --cache-max-mb 100 \
  --dedupe-cap 200000

⁠Command Options

⁠Required

Flag	Description	Example
`--input`	Path to your Nginx access log file	`--input /var/log/nginx/access.log`

⁠Output

Flag	Default	Description
`--output`	`enriched.jsonl`	File where enriched data is written

⁠ipgeolocation.io Enrichment

Flag	Description	Example
`--include`	Enable extra enrichment modules (comma-separated). Valid values: `geo_accuracy`, `dma_code`, `user_agent`, `security`, `abuse`, `hostname`, `liveHostname`, `hostnameFallbackLive`	`--include security,abuse`
`--fields`	Return only specific fields (supports nested dot paths)	`--fields location.country_name,asn.organization`
`--excludes`	Remove fields from the response	`--excludes currency,location.country_flag`
`--lang`	Response language (non-English requires paid plans)	`--lang fr`

⁠Caching and Performance

Flag	Default	Description
`--db`	`cache.db`	SQLite cache file path
`--cache-ttl-hours`	`24`	How long cached data is valid (0=disable, -1=never expire, >0=hours)
`--cache-max-mb`	`200`	Maximum size of cache.db on disk (in MB)
`--dedupe-cap`	`200000`	Maximum number of recent IPs remembered per run (controls RAM usage)

⁠API Usage Control

Flag	Description	Example
`--max-enrich`	Maximum number of IPs to enrich	`--max-enrich 500`
`--enrich-all`	Enrich all unique IPs without asking	`--enrich-all`
`--no-prompt`	Disable all interactive questions (recommended for Docker, CI, scripts)	`--no-prompt`

⁠Parallelism and Rate Limiting

Flag	Default	Description
`--workers`	`10`	Number of parallel workers for concurrent API requests
`--rps`	`5`	Maximum API requests per second (0 = unlimited, recommended to avoid rate limits)

⁠Important Notes:

When --no-prompt is used, the tool will never pause for user input. If neither --max-enrich nor --enrich-all is provided, the tool will automatically enrich all unique IPs.
The --workers flag controls how many concurrent goroutines process IPs simultaneously, improving throughput for large log files.
The --rps flag prevents hitting API rate limits by controlling the maximum requests per second across all workers. Set to 0 for unlimited (use with caution).
Increasing --workers improves performance but may require a higher --rps limit to avoid throttling.

⁠What You Get

Each IP in your logs is enriched with:

Location: Country, city, region, coordinates, timezone
Network: ISP, organization, hosting provider, connection type
ASN: Autonomous system number, name, allocation, routes
Security: VPN/proxy/TOR detection, bot identification, threat scores (requires --include security)
Abuse Contacts: Email and organization for abuse reports (requires --include abuse)
Hostname: Reverse DNS lookup (requires --include hostname, liveHostname, or hostnameFallbackLive)
User Agent: Parsed browser/OS/device info (requires --include user_agent)
Geo Accuracy: Accuracy radius, confidence, locality (requires --include geo_accuracy)
DMA Code: Nielsen DMA code for US IPs (requires --include dma_code)
Cultural Info: Currency, language, calling codes

⁠Output Structure

The response object has several top-level keys. location, network, asn, company, and currency are always present. Optional modules (security, abuse, hostname, time_zone, user_agent) appear only when requested via --include.

{
  ip
  cached
  data {
    ip
    location { ... }       ← always present
    network { ... }        ← always present
    asn { ... }            ← always present (top-level sibling of network)
    company { ... }        ← always present (top-level sibling of network)
    currency { ... }       ← always present
    country_metadata { ... }
    time_zone { ... }      ← always present
    hostname              ← present with --include hostname / liveHostname / hostnameFallbackLive
    security { ... }       ← present with --include security
    abuse { ... }          ← present with --include abuse
    user_agent { ... }     ← present with --include user_agent
  }
}

Field paths for --fields: Because asn and company are top-level siblings of network, use asn.organization, not network.asn.organization.

⁠Output Example

The example below shows a fully-enriched response with all optional modules enabled (--include security,abuse,hostname,user_agent,geo_accuracy,dma_code). Fields that require a specific --include value are annotated.

{
  "cached": false,
  "ip": "8.8.8.8",
  "data": {
    "ip": "8.8.8.8",
    "location": {
      "continent_code": "NA",
      "continent_name": "North America",
      "country_code2": "US",
      "country_code3": "USA",
      "country_name": "United States",
      "country_name_official": "United States of America",
      "country_capital": "Washington, D.C.",
      "country_emoji": "🇺🇸",
      "country_flag": "https://ipgeolocation.io/static/flags/us_64.png",
      "is_eu": false,
      "state_prov": "California",
      "state_code": "US-CA",
      "district": "Santa Clara",
      "city": "Mountain View",
      "zipcode": "94043-1351",
      "latitude": "37.42240",
      "longitude": "-122.08421",
      "geoname_id": "6301403",
      "accuracy_radius": "21.258",
      "confidence": "low",
      "locality": "Mountain View",
      "dma_code": "807"
    },
    "country_metadata": {
      "calling_code": "+1",
      "languages": ["en-US", "es-US", "haw", "fr"],
      "tld": ".us"
    },
    "time_zone": {
      "name": "America/Los_Angeles",
      "offset": -8,
      "offset_with_dst": -7,
      "current_time": "2025-01-15 10:30:00.000-0700",
      "current_time_unix": 1736955000,
      "is_dst": true,
      "dst_savings": 1
    },
    "network": {
      "isp": "Google LLC",
      "connection_type": ""
    },
    "asn": {
      "as_number": "AS15169",
      "asn_name": "GOOGLE",
      "organization": "Google LLC",
      "country": "US",
      "rir": "ARIN",
      "type": "BUSINESS",
      "domain": "google.com",
      "date_allocated": "2012-02-24T00:00",
      "allocation_status": "",
      "num_of_ipv4_routes": "1013",
      "num_of_ipv6_routes": "104"
    },
    "company": {
      "name": "Google LLC",
      "domain": "google.com",
      "type": "Hosting"
    },
    "currency": {
      "code": "USD",
      "name": "US Dollar",
      "symbol": "$"
    },
    "hostname": "dns.google",
    "security": {
      "threat_score": 0,
      "is_tor": false,
      "is_proxy": false,
      "proxy_type": "",
      "is_anonymous": false,
      "is_known_attacker": false,
      "is_spam": false,
      "is_bot": false,
      "is_cloud_provider": true,
      "cloud_provider_name": "Google Cloud"
    },
    "abuse": {
      "route": "8.8.8.0/24",
      "country": "US",
      "handle": "ABUSE5250-ARIN",
      "email": "[email protected]",
      "name": "Abuse",
      "organization": "Google LLC"
    },
    "user_agent": {
      "user_agent_string": "Mozilla/5.0 ...",
      "name": "Chrome",
      "type": "Browser",
      "version": "120.0.0",
      "version_major": "120",
      "device": {
        "name": "Desktop",
        "type": "Desktop",
        "brand": "",
        "cpu": "Intel"
      },
      "operating_system": {
        "name": "Windows",
        "type": "Desktop",
        "version": "10"
      }
    }
  }
}

⁠Notes on optional fields

Field	Requires `--include`	Notes
`location.accuracy_radius`	`geo_accuracy`	Approximate radius of geolocation estimate
`location.confidence`	`geo_accuracy`	Confidence level of the city-level fix
`location.locality`	`geo_accuracy`	Granular locality within a city
`location.dma_code`	`dma_code`	Nielsen DMA code; US IPs only
`hostname`	`hostname` / `liveHostname` / `hostnameFallbackLive`	Reverse DNS; `liveHostname` queries DNS live, `hostnameFallbackLive` falls back to live lookup if cached hostname is unavailable
`security`	`security`	VPN/proxy/TOR/bot detection and threat scoring
`abuse`	`abuse`	Abuse contact email and organization
`user_agent`	`user_agent`	Parsed browser, OS, and device data

⁠Docker-Specific Notes

⁠Volume Mounting

All file paths inside the container must use the /data/ prefix:

✅ Correct: --input /data/access.log
❌ Wrong: --input access.log

⁠Environment Variables

Set your API key via -e flag:

-e IPGEOLOCATION_API_KEY="your_key_here"

⁠Persistent Cache

The cache database persists on your host machine, so subsequent runs are faster:

# First run: slow (API calls)
docker run ... enrich --input /data/access.log
 
# Second run: fast (90%+ cache hits)
docker run ... enrich --input /data/access.log

⁠File Permissions

Output files are created with the container's user permissions. If you need specific ownership:

docker run --rm -it --user $(id -u):$(id -g) \
  -e IPGEOLOCATION_API_KEY="YOUR_API_KEY" \
  -v "$PWD:/data" \
  devjfreaks/nginx-logs-enricher:latest \
  enrich --input /data/access.log

⁠Performance Characteristics

Log Size	Unique IPs	Configuration	First Run	Cached Run
100MB	5,000	Default (10 workers, 5 RPS)	~17 min	~2 min
1GB	50,000	High (20 workers, 20 RPS)	~42 min	~5 min
10GB	500,000	Maximum (30 workers, 30 RPS)	~5 hours	~30 min

⁠Support & Documentation

Built and maintained by https://ipgeolocation.io⁠

Tag summary

Recent tags

Content type

Image

Digest

sha256:611008ddc…

Size

8 MB

Last updated

8 months ago

docker pull devjfreaks/nginx-logs-enricher