Automated scraper for ARK Invest trades, publishing data to a message queue for downstream processin
817
Scrapes the daily ARK Invest trades file, parses each row, and publishes trades to RabbitMQ, with optional exception emails on failures. ARK Invest publishes trades once per day, but this project does not include an internal scheduler. The suggested and tested approach is running it as a Kubernetes CronJob.
| Variable | Required | Default | Description |
|---|---|---|---|
ARK_TRADE_FILE_URL | yes | - | URL to the ARK trades XLS file. |
RABBITMQ_URL | yes | - | RabbitMQ connection URL. |
RABBITMQ_VHOST | no | - | RabbitMQ virtual host (appended to RABBITMQ_URL if URL has no vhost). |
RABBITMQ_USERNAME | no | - | RabbitMQ username (applied to RABBITMQ_URL if URL has no credentials). |
RABBITMQ_PASSWORD | no | - | RabbitMQ password (applied to RABBITMQ_URL if URL has no credentials). |
RABBITMQ_EXCHANGE | yes | - | Exchange to publish trades to. |
RABBITMQ_QUEUE | yes | - | Queue to bind and publish into. |
RABBITMQ_ROUTING_KEY | yes | - | Routing key for published messages. |
TO_EMAILS | no | - | Comma-separated list of emails for exception alerts (emailing is disabled unless this and MAILER_API_URL are set). |
MAILER_API_URL | no | - | REST endpoint that sends the exception email (emailing is disabled unless this and TO_EMAILS are set). |
MAILER_API_KEY | no | - | Optional API key for the mailer service (sent as Authorization: Bearer ...). |
MAILER_INCLUDE_TRACEBACK | no | false | When true, include full tracebacks in exception emails. |
This image will be published on Docker Hub.
docker pull <your-dockerhub-username>/ark-scraper-collector:latest
docker run --rm \
-e ARK_TRADE_FILE_URL="https://example.com/path/to/ark_trades.xlsx" \
-e RABBITMQ_URL="amqp://guest:guest@rabbitmq/" \
-e RABBITMQ_EXCHANGE="ark.trades" \
-e RABBITMQ_QUEUE="ark.trades" \
-e RABBITMQ_ROUTING_KEY="ark.trades" \
<your-dockerhub-username>/ark-scraper-collector:latest
If you use any IDE that supports devcontainers, open the repo in the provided devcontainer. It includes the right Python version and dependencies so you can start coding without extra setup.
For local work outside the container:
uv sync --group dev
Run the test suite with:
uv run pytest
Content type
Image
Digest
sha256:d9f6d1a73…
Size
88.4 MB
Last updated
10 months ago
docker pull schwitzd/ark-scraper-collector