rocketdude147/comicvinecacher repository overview

⁠comicvinecacher

A caching, rate-limit-smoothing drop-in replacement for the Comic Vine API.

Point any Comic Vine client at it instead of comicvine.gamespot.com and it serves from a PostgreSQL cache, falling through to the real API only when it must — and when it does, it spreads those calls across up to 20 independent upstreams, each one an API key that can exit through its own VPN.

The point is not latency. Comic Vine's limit is per-key and per-IP, and a full library walk exhausts it in minutes. One key behind one IP gets throttled; nine key+IP pairs behind a coordinator do not.

Mylar3 / any Comic Vine client
        │  http://host:15650/api/...
        ▼
   comicvine  ──►  PostgreSQL   (10 tables: cache, search_index, image_cache, …)
   coordinator ─►  /data/image_cache   (cover art on disk)
        │
        ├─► cv_vpn_1 ──► comicvine.gamespot.com   key 1, exit NL
        ├─► cv_vpn_2 ──► comicvine.gamespot.com   key 2, exit US
        └─► … up to 20

⁠Tags

Image is python:3.11-slim + postgresql-client-16 (needed for backup/restore, which shell out to pg_dump/psql). Served by gunicorn on port 5000, 1 worker × 16 threads.

The worker count is pinned at 1 deliberately — the rate limiter, the upstream health state and the stats cache are all in-process. Running 2 workers gives you two independent rate limiters that each think they own your full quota.

⁠Prerequisites

1. PostgreSQL 13+, reachable from the container. It need not be in this stack. Create an empty database and an owning user — the app creates all 10 tables itself on first boot: cache, search_index, image_cache, api_keys, clients, settings, metrics, request_log, activity_log, crawl_progress.
2. One or more Comic Vine API keys — https://comicvine.gamespot.com/api/⁠
3. A WireGuard VPN provider, only if you want more than one upstream. The reference stack uses AirVPN via gluetun⁠.
4. /dev/net/tun on the host — only for the VPN sidecars.

⁠Minimal setup — one key, no VPN

services:
  comicvine:
    image: rocketdude147/comicvinecacher:test
    container_name: comicvine
    restart: unless-stopped
    ports:
      - "15650:5000"
    volumes:
      - comicvine-data:/data
      - ./image_cache:/data/image_cache
      - ./backup:/backup
    environment:
      - DB_HOST=192.168.1.10
      - DB_PORT=5432
      - DB_NAME=comicvine
      - DB_USER=comicvine
      - DB_PASSWORD=CHANGE_ME
      - API_KEY_1=YOUR_COMIC_VINE_KEY
      - CACHE_IMAGES=true
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:5000/api/health"]
      interval: 120s
      timeout: 10s
      retries: 3

volumes:
  comicvine-data:

docker compose up -d
curl http://localhost:15650/api/health

Dashboard at http://localhost:15650/⁠.

⁠Full setup — nine upstreams behind nine VPN exits

Each upstream is an API_KEY_n + PROXY_URL_n + UPSTREAM_n_NAME triple. Pair each key with a different exit country — the limit is per-IP as well as per-key, so nine keys sharing one exit buys almost nothing.

⁠Coordinator

services:
  comicvine:
    image: rocketdude147/comicvinecacher:test
    container_name: comicvine
    restart: unless-stopped
    depends_on:
      vpn_1: { condition: service_healthy }
      vpn_2: { condition: service_healthy }
      # …through vpn_9
    ports:
      - "15650:5000"
    cpus: 1.0
    mem_limit: 1g
    volumes:
      - comicvine-data:/data
      - /srv/comicvine/image_cache:/data/image_cache
      - /srv/comicvine/backup:/backup
    environment:
      - DB_HOST=192.168.1.10
      - DB_PORT=5432
      - DB_NAME=comicvine
      - DB_USER=comicvine
      - DB_PASSWORD=CHANGE_ME
      - DB_MAX_CONN=25
      - DATA_DIR=/data

      - INSTANCE_ID=coordinator
      - INSTANCE_NAME=Coordinator-LB
      - INSTANCE_MODE=standalone
      - LOAD_BALANCER_ENABLED=true

      - API_KEY_1=KEY_ONE
      - PROXY_URL_1=http://cv_vpn_1:8888
      - UPSTREAM_1_NAME=A-NL

      - API_KEY_2=KEY_TWO
      - PROXY_URL_2=http://cv_vpn_2:8888
      - UPSTREAM_2_NAME=A-US

      # …to API_KEY_9 / PROXY_URL_9 / UPSTREAM_9_NAME

      - CACHE_IMAGES=true
      - DEBUG=false
      - REQUESTS_PER_HOUR=200
      - MIN_REQUEST_INTERVAL=0.4
      - CRAWL_DELAY=1.5
    networks: [comicvine_lb]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:5000/api/health"]
      interval: 120s
      timeout: 10s
      retries: 3

⁠One VPN sidecar — repeat ×9

  vpn_1:
    image: qmcgaw/gluetun:latest
    container_name: cv_vpn_1              # must match PROXY_URL_1
    restart: unless-stopped
    cap_add: [NET_ADMIN]
    sysctls:
      - net.ipv4.conf.all.src_valid_mark=1
    devices:
      - /dev/net/tun:/dev/net/tun
    ports:
      - "8881:8888"                       # optional, for debugging only
    environment:
      - VPN_SERVICE_PROVIDER=airvpn
      - VPN_TYPE=wireguard
      - WIREGUARD_PRIVATE_KEY=YOUR_WG_PRIVATE_KEY
      - WIREGUARD_PRESHARED_KEY=YOUR_WG_PRESHARED_KEY
      - WIREGUARD_ADDRESSES=10.128.0.2/32
      - SERVER_COUNTRIES=Netherlands      # different per sidecar
      - HTTPPROXY=on
      - HTTPPROXY_LOG=off
      # Required. Without it gluetun's firewall drops the Docker bridge and the
      # coordinator can never reach the proxy — every upstream reports dead.
      - FIREWALL_OUTBOUND_SUBNETS=192.168.0.0/16,172.16.0.0/12,10.0.0.0/8
    volumes:
      - gluetun_1:/gluetun
    networks: [comicvine_lb]
    healthcheck:
      test: ["CMD-SHELL", "wget -q --spider http://10.128.0.1 || exit 1"]
      interval: 60s
      timeout: 15s
      retries: 3
      start_period: 60s
    cpus: 3
    mem_limit: 512m

networks:
  comicvine_lb:
    driver: bridge

volumes:
  comicvine-data:
  gluetun_1:
  # gluetun_2 … gluetun_9

⁠How LB mode turns itself on

Read this before debugging why load balancing is or is not active. From app.py:

LOAD_BALANCER_MODE = os.environ.get('LOAD_BALANCER_ENABLED','false').lower() == 'true' \
                     or any(os.environ.get(f'API_KEY_{i}')       for i in range(1,10)) \
                     or any(os.environ.get(f'UPSTREAM_{i}_KEY')  for i in range(1,10))

Consequences:

• Setting any of API_KEY_1…API_KEY_9 enables LB mode. LOAD_BALANCER_ENABLED=false does not turn it back off.
• Detection only scans indices 1–9, but the upstream loader scans 1–19. So keys at API_KEY_10+ are used once LB mode is on, but cannot switch it on by themselves. If you are running more than nine, keep at least one in the 1–9 range or set LOAD_BALANCER_ENABLED=true explicitly.
• UPSTREAM_n_KEY / UPSTREAM_n_PROXY are accepted as aliases for API_KEY_n / PROXY_URL_n. UPSTREAM_n_KEY wins if both are set.
• ASSIGNED_API_KEY or API_KEY_COORDINATOR adds one more upstream that exits directly, with no proxy, labelled Coordinator (Direct). Useful as a fallback; note it uses your host IP.

⁠Environment reference

Every variable the code actually reads, with its real default.

⁠Database — app.py

⁠Paths

⁠Rate limiting — per upstream, not total

app_loadbalancer.py carries its own lower defaults (REQUESTS_PER_HOUR=180, MIN_REQUEST_INTERVAL=2.0) for the per-upstream limiter. Set both explicitly if the values matter to you, rather than relying on whichever applies.

⁠Crawler

⁠Behaviour

⁠Server

⁠Instance / distributed — app_distributed.py

Only relevant if you run coordinator + worker instances. INSTANCE_MODE=standalone (the default) keeps all of this inert.

⁠Using it as a Comic Vine replacement

Set your client's base URL to http://<host>:15650/api. Request and response shapes match the real API, so nothing else changes.

⁠Passthrough endpoints

Singular and plural both work (issue and issues) — plural is canonical, and singular is normalised via a lookup table.

Resource types supported (20): issues, volumes, characters, publishers, story_arcs, teams, people, concepts, locations, objects, origins, powers, movies, promos, episodes, series_list, videos, video_types, video_categories, video_shows.

Anything else returns Comic Vine's own error shape — status_code: 102, HTTP 400.

⁠Management endpoints

Pre-warm the cache before a big import:

curl -X POST http://localhost:15650/api/crawl/start \
     -H 'Content-Type: application/json' \
     -d '{"resource_types":["volumes","issues"],"resume":true}'

curl http://localhost:15650/api/crawl/status

⁠Gotchas

FIREWALL_OUTBOUND_SUBNETS is mandatory. gluetun firewalls everything outside the tunnel by default, including the Docker bridge. Omit it and every upstream sits permanently unhealthy with no useful error.

depends_on: service_healthy matters. gluetun needs 30–60 s. Without the gate the coordinator starts first, marks every upstream dead, and needs a manual restart.

LOAD_BALANCER_ENABLED=false does not disable LB mode if any API_KEY_1..9 is set.

REQUESTS_PER_HOUR is per upstream. Nine upstreams at 200 is ~1,800/hour total. Setting it to a huge number does not raise Comic Vine's limit; it only stops this app from protecting you from it.

Do not raise gunicorn workers above 1. Rate limiter, upstream health and stats cache are all in-process.

postgresql-client-16 is in the image for a reason. Backup and restore shell out to pg_dump/psql; a major-version mismatch with your server will fail at restore time.

Mount /backup somewhere real before running /api/backup/create.

PROXY_MODE=false makes it cache-only — misses return empty rather than hitting Comic Vine. That is the offline mode; it is not the default.

⁠Security

The reference compose holds nine API keys, WireGuard private keys and a database password. Treat it as a secret — use .env or Docker secrets:

    environment:
      - API_KEY_1=${CV_API_KEY_1}
      - WIREGUARD_PRIVATE_KEY=${WG_KEY_1}

This service performs no authentication of its own. Anyone who can reach port 15650 can spend your rate limit and read your cache. Keep it on the LAN.