Public | Automated Build

Last pushed: 2 years ago
Short Description
logstash is a tool for managing events and logs
Full Description

Logstash Dockerfile

This is a highly configurable logstash (1.4.5) image running Elasticsearch (1.7.0) and Kibana (3.1.2).

How to use this image

To run the image, you have to first decide which services you want to run in your container:

Full ELK stack (default)

$ docker run -d \
  -p 9292:9292 \
  -p 9200:9200 \
  pblittle/docker-logstash

Logstash agent only

$ docker run \
  pblittle/docker-logstash \
  agent

Logstash config test only

$ docker run \
  pblittle/docker-logstash \
  configtest

Embedded Kibana web interface only

$ docker run \
  pblittle/docker-logstash \
  web

Logstash configuration

There are currently two supported ways of including your Logstash config files in your container:

  • Download your config files from the Internet
  • Mount a volume on the host machine containing your config files

Any files in /opt/logstash/conf.d with the .conf extension will get loaded by logstash.

Download your config files from the Internet

To use your own hosted config files, your config files must be one of the following two file types:

  • A monolithic config file (*.conf)
  • A tarball containing your config files (*.tar, *.tar.gz, or *.tgz)

With your config files ready and in the correct format, set LOGSTASH_CONFIG_URL to your logstash config URL using the -e flag as follows:

$ docker run -d \
  -e LOGSTASH_CONFIG_URL=<your_logstash_config_url> \
  -p 9292:9292 \
  -p 9200:9200 \
  pblittle/docker-logstash

By default, if LOGSTASH_CONFIG_URL isn't defined, an example logstash.conf will be downloaded and used in your container.

The default logstash.conf only listens on stdin and file inputs. If you wish to configure tcp and/or udp input, use your own logstash configuration files and expose the ports yourself. See logstash documentation for config syntax and more information.

Mount a volume containing your config files

To use config files from the local file system, mount the config directory as a volume using the -v flag. For example:

$ docker run -d \
  -v <your_logstash_config_dir>:/opt/logstash/conf.d \
  -p 9292:9292 \
  -p 9200:9200 \
  pblittle/docker-logstash

Elasticsearch server integration

If you plan on using Elasticsearch, the following three integration methods are supported:

  • A linked container running Elasticsearch
  • An external Elasticsearch server
  • The embedded Elasticsearch server

Linked container running Elasticsearch

If you want to link to a container running Elasticsearch, simply use the --link flag to connect to the container:

$ docker run -d \
  --link <your_es_container_name>:es \
  -p 9292:9292 \
  pblittle/docker-logstash

To have the linked Elasticsearch container's bind_host and port automatically detected, you will need to set the bind_host and port to ES_HOST and ES_PORT respectively in your elasticsearch output config. For example:

output {
  elasticsearch {
    bind_host => "ES_HOST"
    port => "ES_PORT"
    protocol => "http"
  }
}

If you are linking to an Elasticsearch container running on 172.0.4.20:9200, the config above will be transformed into:

output {
  elasticsearch {
    host => "172.0.4.20"
    port => "9200"
    protocol => "http"
  }
}

External Elasticsearch server

If you are using an external Elasticsearch server, simply set the ES_HOST and ES_PORT environment variables in your run command:

$ docker run -d \
  -e ES_HOST=<your_es_service_host> \
  -e ES_PORT=<your_es_service_port> \
  -p 9292:9292 \
  pblittle/docker-logstash

Embedded Elasticsearch server

The embedded Elasticsearch server will be used by default if you don't provide either of the configuration options above.

Please note, the embedded Elasticsearch server was not designed for use in Production.

Optional, build and run the image from source

If you prefer to build from source rather than use the pblittle/docker-logstash trusted build published to the public Docker Registry, execute the following:

$ git clone https://github.com/pblittle/docker-logstash.git
$ cd docker-logstash

If you are using Vagrant, you can build and run the container in a VM by executing:

$ vagrant up
$ vagrant ssh
$ cd /vagrant/1.4

From there, build and run a container using the newly created virtual machine:

$ make

Finally, verify the installation

You can now verify the logstash installation by visiting the sample Kibana dashboard:

http://<your_container_ip>:9292/index.html#/dashboard/file/default.json

Thank you

A huge thank you to the project Contributors and users. I really appreciate the support.

Contributing

  1. Fork it
  2. Checkout the develop branch (git checkout -b develop)
  3. Create your feature branch (git checkout -b my-new-feature)
  4. Commit your changes (git commit -am 'Add some feature')
  5. Push to the branch (git push origin my-new-feature)
  6. Create new Pull Request

License

This application is distributed under the Apache License, Version 2.0.

Docker Pull Command
Owner
pblittle
Source Repository

Comments (38)
painhardcore
2 years ago

Who can update this image to 1.5.3 with 4 kibana?

pblittle
2 years ago

@beh01der thank you for sharing the flaws that you have found in my project. I'm sorry it is causing you so much pain. I suggest you either don't use the project, or, create a Github issue starting a constructive discussion.

beh01der
3 years ago

I believe a docker image must be self-contained. That's the point of it. Once it's built it must not go to the internet to grab some configs or some other stuff. Well, at least not by default. Running this image is really painful when you are behind a proxy.

What makes it worse is defining a startup script with ENTRYPOINT. It must be defined with CMD to make it easier to start container in interactive mode for debugging.

Also, it's a good practice to keep instructions in Dockerfile to minimum. All RUN commands can be combined into one. ADD command normally goes before RUN.

pblittle
3 years ago

@engendro, sorry, I didn't mean to leave you hanging. Do you mind creating a Github issue with your question? I'll be glad to work through the issue with you.

engendro
3 years ago

Hi, this is an awesome container. I'm currently having an issue. If I try to test the kibana url from outside of the container host. I get an error in the kibana screen: Error Could not reach http://127.0.0.1:9200/_nodes. If you are using a proxy, ensure it is configured correctly. Do you know what is missing to me?. I'm running with the ES embedded server configuration... Thanks!

pblittle
3 years ago

@blackkitten, not a silly question at all. Thanks for the kind words. Regarding the ability to use a different version of Kibana, I would recommend running Kibana in a separate container and have that container connect to this project.

As an aside, if you're interested in testing different configurations, I have a beta tag that is almost ready to ship. It will store your logstash configs in /opt/logstash/conf.d and allow you to have directories of configs rather than one logstash.conf file. Your configs will be downloaded from the internet, decompressed and moved to /opt/logstash/conf.d .

The code is in the develop branch on github [1]. Please create an issue if you do play with it and have any comments or questions.

[1] https://github.com/pblittle/docker-logstash/tree/develop

blackkitten
3 years ago

Hi, a (maybe silly) question from a newbie: this container is wonderful form my test purposes, is it possible to "upgrade" kibana to a newer version?
Can you recommend me a good strategy?
Currently I don't care about persisting the changes to the container, maybe this can simplify things...
Thanks!!

pblittle
3 years ago

@khebbie, thanks for the comment. I have created a GitHub issue to restrict outside access to 9200 and 9300.

khebbie
3 years ago

Please note that versions of elastisearch before 1.2 had a sever vulnerability.
So please remember to not expose it to the internet.
I had mine exposed for a few hours, which was enough to let bad guys in .
http://security.stackexchange.com/questions/58862/logging-server-compromised-iptables-and-iptablex

pblittle
3 years ago

@fernandodbc, you can add your logstash-forwarder SSL keys to the container by setting the LF_SSL_CERT_KEY_URL and LF_SSL_CERT_URL variables in your docker environment. [1]

Please let me know if I can do anything else to help.

[1] https://github.com/pblittle/docker-logstash/blob/master/bin/boot#L41