qq58945591/logstash

By qq58945591

•Updated almost 9 years ago

logstash for docker (base from alpine linux)

Image

1.1K

Overview Tags

qq58945591/logstash repository overview

⁠关于本镜像:

自定义logstash配置的docker image, 增加了配置nginx与postfix的自定义日志格式匹配..
增加了geoip数据库。可以方便调用ip数据库, 在logstash 5.0之后使用的数据库是GeoLite2-City.mmdb.
使用alpine linux作为系统底层, 减小image体积.

⁠如何使用?

拉取镜像.

docker pull qq58945591/logstash

启动logstash, 默认输出到elaticsearch.

docker run -d --name logstash qq58945591/logstash -f /etc/logstash/conf.d/logstash.conf

⁠或许你想自定义配置? 使用Dockerfile.

编辑一个Dockerfile如下内容:

FROM logstash

COPY logstash.conf /some/config-dir/

CMD ["-f", "/some/config-dir/logstash.conf"]

然后执行build

docker build -t my-logstash .

⁠ELK 组合

附带一个elk_example.yml, 可以快速建立一个elk日志收集套件，默认只对nginx与postfix进行处理,想要处理更多日志，也许需要自定义logstash的Dockerfile以及过滤配置.

⁠elk_example.yml

version:  "2"
services:
  elasticsearch:
    image:  qq58945591/elasticsearch
    ports:
      - 127.0.0.1:9200:9200
      - 127.0.0.1:9300:9300
    volumes:
      - /opt/elasticsearch/data:/usr/share/elasticsearch/data
    user: elasticsearch
    restart:  always
    environment:
      - Enode.name=elk_node1
      - Ecluster.name=mycluster
    container_name: "elasticsearch"
    networks:
      - elk_network

  logstash:
    image:  qq58945591/logstash
    user: root
    restart:  always
    links:
      - redis:redis-server
      - elasticsearch:elasticsearch
    command: "-f /etc/logstash/conf.d/logstash.conf"
    container_name: "logstash"
    depends_on:
      - elasticsearch
      - redis
    ulimits:
      mem_limit: 1024m
    networks:
      - elk_network

  kibana:
    image:  qq58945591/kibana
    ports:
      - 127.0.0.1:5601:5601
    user: root
    restart:  always
    links:
      - elasticsearch:elasticsearch
#    environment:
#      - ELASTICSEARCH_URL=http://elasticsearch:9200
    container_name: "kibana"
    depends_on:
      - elasticsearch
    ulimits:
      mem_limit: 512m
    networks:
      - elk_network

  redis:
    image:  redis
    ports:
      - 127.0.0.1:6379:6379
    volumes:
      - /opt/redis:/data
    user: root
    restart:  always
    container_name: "redis-server"
    ulimits:
      mem_limit: 1024m
    networks:
      - elk_network

networks:
  elk_network:
     driver: bridge
     driver_opts:
       com.docker.network.enable_ipv6: "false"
     ipam:
       driver: default
       config:
         - subnet: 172.18.1.0/24
           gateway: 172.18.1.1

⁠注意：

由于使用redis作为broker, 日志的收集流程是rsyslog(客户端节点) --> rsyslog(中央日志服务器) --> redis(broker缓冲队列) --> logstash(结构化处理) --> elaticsearch(存储) ,默认情况下rsyslog不支持往redis推送数据,需要重编译rsyslog,添加--enable-omhiredis参数,详情参考这里Redis Output Module⁠ 这里有一个实例可以参考Recipe: rsyslog + Redis + Logstash⁠

⁠注意：

在中央日志服务器修改/etc/rsyslog.conf

添加下面内容,开启输出到redis模块.

#Provides outpu to Redis
module(load="omhiredis")

增加一个输出模版.

template(name="json_lines" type="list" option.json="on") {
constant(value="{")
         constant(value="\"timestamp\":\"")      property(name="timereported" dateFormat="rfc3339")
         constant(value="\",\"message\":\"")     property(name="rawmsg-after-pri")
         constant(value="\",\"host\":\"")        property(name="hostname")
         constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
         constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
         constant(value="\",\"program\":\"")   property(name="syslogtag")
       constant(value="\"}")
     }

添加主队列设定

main_queue(
  queue.workerthreads="2"      # threads to work on the queue
  queue.dequeueBatchSize="200" # max number of messages to process at once
  queue.size="10000"           # max queue size
)

光标移动到rsyslog.conf最后,加入过滤条件然后传送给logstash处理,更多条件过滤规则清参考Syslog Filter Conditions⁠

只传送包含发送状态的日志.

if $syslogtag contains 'postfix' and $rawmsg-after-pri contains 'status=' and not ($msg contains 'connect from' and $msg contains 'disconnect from' or $msg contains 'dsn-feeder-prod' or $msg contains 'root') then {

action (
        name="push_postfix_to_redis"
        server="127.0.0.1"
        serverport="6379"
        type="omhiredis"
        mode="queue"
        key="mail"
        template="json_lines"
)
}

只传送包含tag 为nginx以及不包含zabbix内容的web记录,

if $syslogtag contains 'nginx' and not ($msg contains "Zabbix" ) then {

action (
        name="push_nginx_to_redis"
        server="127.0.0.1"
        serverport="6379"
        type="omhiredis"
        mode="queue"
        key="nginx"
        template="json_lines"
)
}

使用docker-compose启动ELK

docker-compose -f /path/to/path/elk_example.yml up -d

检查容器运行状态

docker logs 容器名或ID -f

日志收集节点只需要设定将日志转发到中央服务器即可邮件日志在节点客户端/etc/rsyslog.conf

mail.* @@rsyslog.server:514

nginx日志仅需要在nginx.conf或者单独server配置发送到中央日志服务器即可注意tag标志一定要与远程日志服务器对应. 参考nginx官方文档⁠

access_log syslog:server=rsyslog.server:514,facility=local6,tag=nginx,severity=info main;

Tag summary

Recent tags

Content type

Image

Digest

Size

186.1 MB

Last updated

almost 9 years ago

docker pull qq58945591/logstash