Public | Automated Build

Last pushed: 2 hours ago
Short Description
Netdata is a daemon that collects data in realtime (per second)
Full Description

netdata

Dockerfile for building and running a netdata deamon for your host instance.

Netdata monitors your server with thoughts of performance and memory usage, providing detailed insight into very recent server metrics. It's nice, and now it's also dockerized.

More info about project: https://github.com/firehol/netdata

More info about me

I'm primarily a full-stack web developer with strong knowledge of Docker, APIs, AWS, PHP, Go, Nginx+LUA, SQL and NoSQL databases, Video Streaming (Wowza Media Server), and handle DevOps/automation for several large scale international clients (High traffic/HA deployments).

If you need someone with this skillset, please contact me at black@scene-si.org.

I'm also the author of the following books:

For more information, I also write a development-themed blog at scene-si.org.
I occasionally tweet my more plebean pursuits (sometimes in Slovenian) on @TitPetric.

Using

docker run -d --cap-add SYS_PTRACE \
           -v /proc:/host/proc:ro \
           -v /sys:/host/sys:ro \
           -p 19999:19999 titpetric/netdata

Open a browser on http://server:19999/ and watch how your server is doing.

Supported tags and respective Dockerfile links

The tags include builds of netdata, with the same tag in upstream. If there's some need to add older tags, you may
use the provided /releases folder as reference, and add new tags as a PR. The latest tag is in line with the
upstream and is occasionally prone to failure. As far as older tags go - they will inevitably lack some new features
but should provide a more stable version to run.

Developers note: new tags are not added automatically which means there might be some delay between when a new
release of netdata is available and when a new tag is available on docker hub - to add a new release yourself, the procedure is as follows:

  1. fork netdata repo,
  2. run /update-releases.sh,
  3. add, commit, push and submit a PR to titpetric/netdata

When you will submit a PR, I will also add the new version to the docker hub and thank you profusely.

Limiting IP netdata listens to

By default netdata listens to 0.0.0.0 (any address). You might want to change this if you're running netdata in --net=host mode. You can pass the following environment variable:

  • NETDATA_IP - the IP that netdata should listen to, e.g. 127.0.0.1 for localhost only.

Passing custom netdata options

If you need to pass some custom options to netdata, you can pass the following environment variable:

  • NETDATA_ARGS - for example if you don't want to use NETDATA_IP above, you can pass -e NETDATA_ARGS="-i 127.0.0.1" for same effect.

Getting emails on alarms

Netdata supports forwarding alarms to an email address. You can set up sSMTP by setting the following ENV variables:

  • SSMTP_TO - This is the address alarms will be delivered to.
  • SSMTP_SERVER - This is your SMTP server. Defaults to smtp.gmail.com.
  • SSMTP_PORT - This is the SMTP server port. Defaults to 587.
  • SSMTP_USER - This is your username for the SMTP server.
  • SSMTP_PASS - This is your password for the SMTP server. Use an app password if using Gmail.
  • SSMTP_TLS - Use TLS for the connection. Defaults to YES.
  • SSMTP_HOSTNAME - The hostname mail will come from. Defaults to localhost.

For example, using gmail:

-e SSMTP_TO=user@gmail.com -e SSMTP_USER=user -e SSMTP_PASS=password

Alternatively, if you already have s sSMTP config, you can use that config with:

-v /path/to/config:/etc/ssmtp

See the following link for details on setting up sSMTP: SSMTP - ArchWiki

Getting alarms in slack

Netdata supports sending alerts to slack via webhooks. You can set that up by setting the following ENV variables:

  • SLACK_WEBHOOK_URL - This is your incoming slack webhook
  • SLACK_CHANNEL - This is the default channel that alerts will get sent to

For example:

-e SLACK_WEBHOOK_URL=https://hooks.slack.com/services/XXXX -e SLACK_CHANNEL=alerts

Getting alarms in Telegram

Netdata supports sending alerts to Telegram via token and chat ID. You can set that up by setting the following ENV variables:

  • TELEGRAM_BOT_TOKEN - This is your bot token
  • TELEGRAM_CHAT_ID - This is the chat ID

For example:

-e TELEGRAM_BOT_TOKEN=22624413:AAGy12TkSMBYVBTe4lQt3BfUYvUs5h7I1jn -e TELEGRAM_CHAT_ID=137165138

For more details about Telegram alerts, see this page - GitHub

Getting alarms in Pushbullet

Netdata supports sending alerts to Pushbullet via API token. You can set that up by setting the following ENV variables:

  • PUSHBULLET_ACCESS_TOKEN - This is your API token
  • PUSHBULLET_DEFAULT_EMAIL - This is the default email that alerts will get sent to if there is not a Pushbullet account attached to it

For example:

-e PUSHBULLET_ACCESS_TOKEN=o.l8VuizWhXgbERf2Q78ghtzb1LDCYvbSD -e PUSHBULLET_DEFAULT_EMAIL=your.email@gmail.com

More details about Pushbullet alerts are provided here - GitHub

Setting up streaming

On a client netdata set this destination to be the HOST[:PORT] of the
central netdata, and give an API_KEY that is secret and only known internally
to the netdata clients, and netdata central. See this page - GitHub

  • NETDATA_STREAM_DESTINATION - HOST[:PORT] to stream to
  • NETDATA_STREAM_API_KEY - API_KEY to send to central net data
-e NETDATA_STREAM_DESTINATION=netdata.service:19999 -e NETDATA_STREAM_API_KEY=1h213ch12h3rc1289e

On the central netdata set 1 or more NETADATA_API_KEY_ENABLE env variables that matches the API_KEY
that you used on the client above, this will enable the netdata client node to communicate with the netdata central

  • NETADATA_API_KEYENABLE{API_KEY}=1
-e NETDATA_API_KEY_ENABLE_1h213ch12h3rc1289e=1

Monitoring docker container metrics

Netdata supports fetching container data from docker.sock. You can forward it to the netdata container with:

-v /var/run/docker.sock:/var/run/docker.sock

This will allow netdata to resolve container names.

Note: forwarding docker.sock exposes the administrative docker API. If due to some security issue access has been obtained to the container, it will expose full docker API, allowing to stop, create or delete containers, as well as download new images in the host.

TL;DR If you care about security, consider forwarding a secure docker socket with docker-proxy-acl

Monitoring docker notes on some systems (Debian jessie)

On debian jessie only 'cpu' and 'disk' metrics show up under individual docker containers. To get the memory metric, you will have to add cgroup_enable=memory swapaccount=1 to /etc/default/grub, appending the GRUB_CMDLINE_LINUX_DEFAULT variable:

$ cat /etc/default/grub  | grep GRUB_CMDLINE_LINUX_DEFAULT
GRUB_CMDLINE_LINUX_DEFAULT="quiet cgroup_enable=memory swapaccount=1"

After rebooting your linux instance, the memory accounting subsystem of the kernel will be enabled. Netdata will pick up additional metrics for the containers when it starts.

Environment variables

It's possible to pass a NETDATA_PORT environment variable with -e, to start up netdata on a different port.

docker run -e NETDATA_PORT=80 [...]

Some explanation is in order

Docker needs to run with the SYS_PTRACE capability. Without it, the mapped host/proc filesystem is not fully readable to the netdata deamon, more specifically the "apps" plugin:

16-01-12 07:58:16: ERROR: apps.plugin: Cannot process /host/proc/1/io (errno 13, Permission denied)

See the following link for more details: /proc/1/environ is unavailable in a container that is not priviledged

Limitations

In addition to the above requirements and limitations, monitoring the complete network interface list of the host is not possible from within the Docker container. If you're running netdata and want to graph all the interfaces available on the host, you will have to use --net=host mode.

See the following link for more details: network interfaces missing when mounting proc inside a container

Work-around

I provided a script called fakenet.sh which provides a copy of the /proc/net filesystem. You should start this script before you start the netdata container. You can do it like this:

wget https://raw.githubusercontent.com/titpetric/netdata/master/fakenet.sh
chmod a+x fakenet.sh
nohup ./fakenet.sh >/dev/null 2>&1 &

Using the above command, the fakenet script will start in the background and will keep running there. You can use other tools like screen or tmux to provide similar capability.

The script fills out the /dev/shm/fakenet location, which you must mount into the container. You must mount it into /fakenet/proc/net exactly with the option like this:

-v /dev/shm/fakenet:/fakenet/proc/net

The script refreshes network information about every 250ms (four times per second). The interval may be increased to give better accuracy of netdata, but CPU usage will also increase. Because of this, the data is not very accurate and some spikes and valleys will occur because of a shifting window between when the reading was taken (fakeproc) and between when the reading was read by netdata. This means the margin for error is whatever data can be collected in ~250ms.

While the solution might not fit everybody, it's security-positive because the netdata container can only inspect the fake proc/net location, and can't actually access any of the networks because it runs on a private LAN / custom network which is managed and firewalled by docker. You may even open access via application, like a nginx reverse proxy where you can add authentication etc.

Pro/con list:

    • network isolation stays in tact
    • all network device metrics are available
    • one more service to provide fakenet
    • accuracy vs. cpu use is a trade-off

Additional notes

Netdata provides monitoring via a plugin architecture. This plugin supports many projects that don't provide data over the /proc filesystem. When you're running netdata in the container, you will have difficulty providing many of these paths to the netdata container.

What you do get (even with the docker version) is:

  • Host CPU statististics
  • Host Network I/O, QoS
  • Host Disk I/O
  • Applications monitoring
  • Container surface metrics (cpu/disk per name)

You will not get detailed application metrics (mysql, ups, etc.) from other containers or from the host if running netdata in a container. It may be possible to get some of those metrics, but it might not be easy, and most likely not worth it. For most detailed metrics, netdata needs to share the same environment as the application server it monitors. This means it would need to run either in the same container (not even remotely practical), or in the same virtual machine (no containers).

Note: if you have some custom hardware like a UPS which is monitored via USB and netdata supports it, you will most likely need to add new software to the netdata docker image to support it. The correct way to do it is to create your own Dockerfile, start with "FROM titpetric/netdata" and then add all your installation commands to build your own image which will support your hardware setup. Most likely if it's not a very common setup (i.e. available on most machines), the software will not be added to titpetric/netdata - that being said, your use case might be useful for others so feel free to submit issues with your extensions or feature requests in terms of new software. I'll gladly add your project/extension to the README here.

Docker Pull Command
Owner
titpetric
Source Repository

Comments (13)
titpetric
9 months ago

@drmonkey, @marcosatori - the issue has been resolved, more information is in GH issues page. Currently no chowns "hotfixes" apply anymore.

For anyone experiencing issues related to this image, your best bet is to submit an issue report against titpetric/netdata github repo.

marcosartori
10 months ago

I changed run.sh to include this and rebuilt the container

chown -R ${RUN_USER}.${RUN_USER} /usr/share/netdata/ sleep 2s exec /usr/sbin/netdata -D -u ${RUN_USER} -s /host -p ${NETDATA_PORT}

or if your super lazy, you can docker exec into the container and chown the hell out of
chow -R root. /usr/share/netdata/

and then your good to go

dmonkey
10 months ago

Hi, I maintain a template to make it easy to install this docker on unRAID here: https://github.com/Data-Monkey/docker-templates/tree/master/Data-Monkey
There are quite a few people on unRAID using it.
Since the latest update the website does not come up, but instead we get this message:
Access to file '/usr/share/netdata/web/' is not permitted.
I had a look in the docker but I can't see an obvious permission issue.

Any help would be appreciated

titpetric
a year ago

@adolphlwq - seems the options have recently been deprecated. -nd is replaced with -D, -ch is replaced with -s. Thanks to your report I've updated the Dockerfile, and most likely avoided a failure in the near future. Thank you.

Source: https://github.com/firehol/netdata/blob/master/src/main.c#L303
https://github.com/firehol/netdata/blob/master/src/main.c#L308

adolphlwq
a year ago

Hi, I see "CMD /usr/sbin/netdata -nd -ch /host -p ${NETDATA_PORT}" in your Dockerfile.
But when I run /usr/sbin/netdata --help,the output is:
"""
Options:
-c config_file Load alternate configuration file Default: /etc/netdata/netdata.conf
-D Disable fork into background
-h Display help message
-P FILE File to save a pid while running
-i address The IP address to listen to. Default: All addresses
-p port_number Port to listen. Can be from 1 to 65535. Default: 19999
-s PATH Path to access host /proc and /sys when running in a container.
-t seconds The frequency in seconds, for data collection. Same as 'update every' config file option. Default: 1
-u username System username to run as. Default: netdata
-v Version of the program
-W stacksize=<size>|unittest|debug_flag vendor options.
"""
There are no "-n","-d" options.

Can you explain why "CMD /usr/sbin/netdata -nd -ch /host -p ${NETDATA_PORT}" is ok. Thanks.

titpetric
a year ago

@monnierant - https://github.com/firehol/netdata/issues/70 & https://github.com/firehol/netdata/issues/101 - adding http auth is in the road map but it's low prio

titpetric
a year ago

@anhcuong - monitoring docker containers net/cpu has been added in recent versions. File an issue against titpetric/netdata if you believe you have found a bug and are able to reproduce it. If you can add your complete docker run command - that's great.

@monnierant - If you'd like to protect netdata with a user/pass, put a nginx proxy in front of it. You may file an issue against firehol/netdata to add this functionality. You can override the netdata configuration with -v /etc/netdata:/etc/netdata.

monnierant
a year ago

Hi You have done a very nice work with this.
I just wonder. I would like to setup some user and pass for protecting the access. How can i add config files by volumes ?

anhcuong
a year ago

Currently netdata does not support monitoring CPU, memory at container level like cAdvisor right?
Btw, I still have the below problem when running your latest netdata (18Apr2016) even specified the following in run command: --net=host --cap-add SYS_PTRACE
" ERROR: apps.plugin: Cannot process /host/proc/23579/io (errno 13, Permission denied)"

titpetric
a year ago

I stand corrected - it's a known limitation of linux namespaces. More info there: https://github.com/docker/docker/issues/13398

Short answer: if you want to monitor host network interfaces, add --net=host. I've updated the docs to reflect this.