Sign inSign up

anapsix/get-cloudflare-logs

By anapsix

•Updated almost 6 years ago

Image
0

2.0K

anapsix/get-cloudflare-logs repository overview

Retrieving Cloudflare logs via Logpull API⁠ with feron⁠, and pushing them into Elasticsearch with Filebeat⁠.

Artifact HUB Docker Cloud Automated build Docker Pulls Docker Image Size (latest by date)

⁠Helm Chart

The whole shebang can be installed onto K8s cluster with included Helm chart⁠. Chart is published to https://charts.random.io⁠, and registered with Artifact Hub⁠

⁠Local Development

⁠Build
docker build -t get-logs .
⁠Launch

Before launching, make sure to set your CF credentials as environment variables

export CF_ZONE_ID=51e241f08e014feb95d1b2760228d12a
export [email protected]
export CF_AUTH_KEY=51e241f08e014feb95d1b2760228d12a2df50

or modify docker-compose.yaml⁠ appropriately (see docs on env_file⁠, and environment⁠ usage)

After launching local environment, access Kibana via http://localhost:5601/app/kibana#/discover⁠.

⁠With Docker Compose
# (re)build
docker-compose build

# launch Elasticsearch, Kibana, and get-logs container instances
docker-compose up -d

# keep an eye on the logs
docker-compose logs -f get-logs

After launching local environment, access Kibana via http://localhost:5601/app/kibana#/discover⁠.

NOTE: since log collection is running on schedule, data will not appear in ES immediately. Keen an eye on the logs, and ./logs directory. ES index will receive data shortly after you see new file appearing, and / or Harvester started for file:... log message in the logs. Create an index pattern while you are waiting.

⁠Launch manually
# launch Elasticsearch container instance
docker run -d \
  --name es \
  -p 9200:9200 \
  -e "discovery.type=single-node" \
  docker.elastic.co/elasticsearch/elasticsearch:7.6.2

# launch Kibana container instance
docker run -d \
  --name ki \
  -p 5601:5601 \
  --link es:elasticsearch \
  docker.elastic.co/kibana/kibana:7.6.2

# launch Cloudflare Logpull container instance
docker run -it --rm \
  -e CF_AUTH_EMAIL \
  -e CF_AUTH_KEY \
  -e CF_ZONE_ID \
  -e SAMPLE_RATE="0.01" \
  -e ES_HOST="http://elasticsearch:9200" \
  -e ES_INDEX="cloudflare-test" \
  -e ES_INDEX_SHARD=5 \
  -e ES_INDEX_REPLICAS=0 \
  -e ES_INDEX_REFRESH=10s \
  --link es:elasticsearch \
  get-logs

⁠License

Open-sourced software licensed under the MIT license.

⁠Acknowledgments

This repo includes and relies on go-tasks⁠, created by Martin Fabrizzio Vilche⁠. Thank you 🙏, Martin.

Tag summary

Content type

Image

Digest

Size

35.8 MB

Last updated

almost 6 years ago

docker pull anapsix/get-cloudflare-logs