kentik/blueflow

By kentik

Updated about 4 years ago

This image sends flow logs from the IBM Cloud platform into Kentik.

Image

439

This image sends flow logs from the IBM Cloud platform into Kentik.

To learn more about this, see:

Flags exposed by the binary:

Usage of /usr/bin/blueflow:
  -api_email string
    	Kentik Email Address
  -api_root string
    	API url prefix. If not set, defaults to https://api.kentik.com (default "https://api.kentik.com")
  -archive
    	If set, archive the flows with the prefix "processed" instead of deleting after processing
  -auth_endpoint string
    	Auth URL to auth IBM cloud on (default "https://iam.cloud.ibm.com/identity/token")
  -bucket_name string
    	Bucket to find VPC flow logs in.
  -dest string
    	Address to send flow to. If not set, defaults to https://flow.kentik.com (default "https://flow.kentik.com")
  -export_id int
    	Kentik export id
  -healthcheck string
    	Bind to this interface to allow healthchecks
  -log_level string
    	Logging Level (default "debug")
  -metalisten string
    	HTTP port to bind on (default "localhost:0")
  -metrics string
    	Metrics Configuration. none|syslog|stderr|graphite:127.0.0.1:2003 (default "syslog")
  -olly_dataset string
    	Olly dataset name
  -olly_write_key string
    	Olly dataset name
  -plan_id int
    	Kentik Plan ID to use for devices
  -regions string
    	Any comma seperated mix of us-south,us-east,eu-gb,eu-de,jp-tok (default "us-south")
  -redis string
    	host:port string on which to connect to redis
  -redis_db int
    	Redis DB to use
  -sample_rate int
    	Sampling rate to use. 1 -> 1:1 sampling, 2 -> 1:2 sampling and so on. (default 1)
  -service_endpoint string
    	Service URL to connect to IBM cloud on (default "s3.private.us-south.cloud-object-storage.appdomain.cloud")
  -service_name string
    	Service identifier (default "blueflow")
  -shard_follower
    	If true, this process is a sharding follower.
  -shard_leader
    	If true, this process is a sharding leader.
  -site_id int
    	Kentik Site ID to use for devices
  -stdout
    	Log to stdout (default true)
  -v	Show version and build information

This requires the following variables to be set:

  • IBM_VPC_API_KEY
  • IBM_COS_API_KEY
  • IBM_SERVICE_INSTANCE
  • ID of the COS which contains the bucket logs are written to.
  • IBM_BUCKET_NAME
  • Name of the log bucket.
  • IBM_ENDPOINT
  • Endpoint to use to access this bucket.
  • KENTIK_API_TOKEN
  • Token for Kentik's API.
  • KENTIK_EMAIL
  • Email address associated with this token.
  • KENTIK_PLAN_ID
  • Kentik billing plan ID to store flow under.
  • KENTIK_EXPORT_ID
  • Kentik cloud export ID to store flow under.

To run, a basic example is:

docker run             \
       -d              \
       --name blueflow \
       -p 8083:8083 \
       -e IBM_VPC_API_KEY=$IBM_VPC_API_KEY \
       -e KENTIK_API_TOKEN=$KENTIK_API_TOKEN \
       -e IBM_COS_API_KEY=$IBM_COS_API_KEY \
       -e IBM_SERVICE_INSTANCE=$IBM_SERVICE_INSTANCE \
       kentik/blueflow:v1                                             \
       --bucket_name=$IBM_BUCKET_NAME \
       --log_level=info          \
       --service_endpoint=$IBM_ENDPOINT \
       --api_email=$KENTIK_EMAIL \
       --plan_id=$KENTIK_PLAN_ID \
       --export_id=$KENTIK_EXPORT_ID

Verify that this is working and get some useful stats with

curl 127.0.0.1:8083/service/info | jq '.'

A sample docker-compose.yaml file for a more complex setup is:

version: '3'
services:
  redis:
    image: "redis"
    networks:
      blueflow:
        aliases:
          - redis
  blue_leader:
    image: "kentik/blueflow:v1"
    depends_on:
      - redis
    ports:
      - "8083"
    environment:
      - IBM_VPC_API_KEY=$IBM_VPC_API_KEY
      - KENTIK_API_TOKEN=$KENTIK_API_TOKEN
      - IBM_COS_API_KEY=$IBM_COS_API_KEY
      - IBM_SERVICE_INSTANCE=$IBM_SERVICE_INSTANCE
    entrypoint: ["/usr/bin/blueflow", "--bucket_name", "$IBM_BUCKET_NAME", "--log_level", "info", "--service_endpoint", "$IBM_ENDPOINT", "--api_email", "$KENTIK_EMAIL", "--plan_id", "$KENTIK_PLAN_ID", "--export_id", "$KENTIK_EXPORT_ID", "--redis", "redis:6379", "--shard_leader", "--metalisten", "0.0.0.0:8083"]
    networks:
      blueflow:
        aliases:
          - blue_leader
  blue_follower:
    image: "kentik/blueflow:v1"
    depends_on:
      - redis
      - blue_leader
    ports:
      - "8083"
    environment:
      - IBM_VPC_API_KEY=$IBM_VPC_API_KEY
      - KENTIK_API_TOKEN=$KENTIK_API_TOKEN
      - IBM_COS_API_KEY=$IBM_COS_API_KEY
      - IBM_SERVICE_INSTANCE=$IBM_SERVICE_INSTANCE
    entrypoint: ["/usr/bin/blueflow", "--bucket_name", "$IBM_BUCKET_NAME", "--log_level", "info", "--service_endpoint", "$IBM_ENDPOINT", "--api_email", "$KENTIK_EMAIL", "--plan_id", "$KENTIK_PLAN_ID", "--export_id", "$KENTIK_EXPORT_ID", "--redis", "redis:6379", "--shard_follower", "--metalisten", "0.0.0.0:8083", "--sample_rate", "2"]
    networks:
      blueflow:
        aliases:
          - blue_follower

networks:
  blueflow:

Docker Pull Command

docker pull kentik/blueflow