Public Repository

Last pushed: 2 months ago
Short Description
Filebeat container, alternative to fluentd used to ship kubernetes cluster and pod logs
Full Description

filebeat-kubernetes

Filebeat container, alternative to fluentd used to ship kubernetes cluster and pod logs

From: https://github.com/ApsOps/filebeat-kubernetes with little changes:

  • use name in configuration file, for specifying FILEBEAT_HOST
  • change type to k8s-container-logs

Getting Started

This container is designed to be run in a pod in Kubernetes to ship logs to logstash for further processing.
You can provide following environment variables to customize it.


LOGSTASH_HOSTS=logstash:5044,logstash.default.svc:5044
LOG_LEVEL=warning  # log level for filebeat. Defaults to "error".
FILEBEAT_HOST=ip-a-b-c-d # custom "filebeat.name" field. Refer following manifest to set it to k8s nodeName

This should be run as a Kubernetes Daemonset (a pod on every node). Example manifest:


apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: pods-logger
spec:
  template:
    metadata:
      labels:
        name: pods-logger
    spec:
      containers:
      - name: filebeat
        image: bringg/filebeat-kubernetes
        imagePullPolicy: Always
        env:
        - name: LOGSTASH_HOSTS
          value: logstash.default.svc:5044
        - name: LOG_LEVEL
          value: warning
        - name: FILEBEAT_HOST
          valueFrom:
              fieldRef:
                fieldPath: spec.nodeName

        resources:
          limits:
            cpu: 100m
            memory: 256Mi

        volumeMounts:
        - name: var-log-containers
          mountPath: /var/log/containers

        - name: var-log-pods
          mountPath: /var/log/pods
          readOnly: true

        - name: var-lib-docker-containers
          mountPath: /var/lib/docker/containers
          readOnly: true

      volumes:
      - name: var-log-containers
        hostPath: { path: /var/log/containers }

      - name: var-log-pods
        hostPath: { path: /var/log/pods }

      - name: var-lib-docker-containers
        hostPath: { path: /var/lib/docker/containers }

Filebeat parses docker json logs and applies multiline filter on the node before pushing logs to logstash.

Make sure you add a filter in your logstash configuration if you want to process the actual log lines.

filter {
  if [type] == "k8s-container-logs" {

    mutate {
      rename => ["log", "message"]
    }

    date {
      match => ["time", "ISO8601"]
      remove_field => ["time"]
    }

    grok {
        match => { "source" => "/var/log/containers/%{DATA:pod_name}_%{DATA:namespace}_%{GREEDYDATA:container_name}-%{DATA:container_id}.log" }
        remove_field => ["source"]
    }
  }
}

This grok pattern would add the fields - pod_name, namespace, container_name and container id to log entry in Elasticsearch.

Docker Pull Command
Owner
bringg

Comments (0)