panoptes/panoptes_docker repository overview

⁠Panoptes in Docker

A Docker container for testing and observing Panoptes Functionality.

Panoptes is a Python based network telemetry ecosystem that implements discovery, enrichment and polling. To learn more about Panoptes, check out the project's GitHub repository⁠. This repository provides a workbench version of Panoptes inside a Docker container that is intended to be used for testing concepts and observing code operations.

This does not follow best practices for deployment and shouldn't be used for production deployment. It's a very quick and dirty method to get a Panoptes instance up and running in a minimal amount of time with plenty of redundancy in place to allow enterprise scaling. While this isn't an efficient deployment, it does provide an accurate and operational build of the larger structure.

⁠Quick Reference

If you're impatient, and simply want to get this up and running, check out the quickstart guide⁠.

Where to Get Help

Where to file Issues

Issues related to Panoptes_docker⁠

Issues with Panoptes⁠

Image Updates

panoptes_docker PRs⁠

Source of this Description

The panoptes_docker Readme⁠

Maintained by

James Diss: [email protected]⁠

Supported Docker Versions

Tested on 18.09.1⁠

⁠Table of Contents

• Background⁠
• Install⁠
• Usage⁠
• Configuration⁠
• Contribute⁠
• Additional Resources⁠
• License⁠

⁠Background

Panoptes is typically run in a distributed system, but it can be difficult to set up a working hive of the various moving parts. Panoptes can be horizontally scaled for metric collection by adding more capacity to run collection, this docker image simply cuts the total number of nodes down to 1. This version also runs from a json file, but we use a command database internally and are working to extend this to CSV files.

This image⁠ provides a visual representation of the services running inside this container.

This container is based on Ubuntu:18.04, and includes Python2.7, open-jdk 8, redis, zookeeper, and influxdb. There are also a few tools that are included for testing purposes, but are not necessary for operation of Panoptes:

• Netcat - Used to check functionality of the services
• tar - Used to view log archives.
• snmpd - Provides an easy target to poll
• nano - Makes it easier to open and modify configuration files and scripts.
• daemontools - Handles output and schedules the various subcomponents of Panoptes. Panoptes typically relies on its granular logging system to redirect to logging servers, but daemontools solves this requirement here.
• Grafana - Included as a visualization tool. See the section on Grafana⁠ for details.

The container also installs a specific version of Kafka, because of a hard dependency. This is accomplished by downloading and building the proper version from the ASF archives⁠.

The configuration files for the services included in this image are available to browse under the resources/ directory structure. Of particular interest will be the resources/daemontools/panoptes_*.run files, as these are the calls to the various Panoptes components.

⁠Install and Setup

First, you will need to have docker installed on your machine⁠.

Then, You can either run the pre-built image we've provided on Docker Hub⁠, or Build the image yourself from the source code⁠.

To use the pre-built image from Docker Hub, simply pull it with docker pull:

docker pull panoptes/panoptes_docker

To build the image from scratch, first clone the repository and run docker build from inside the directory

git clone https://github.com/yahoo/panoptes_docker.git && cd panoptes_docker
docker build . -t panoptes_docker

⁠Usage

This container requires quite a lot of processing power since it contains a distributed system that has been packed into a small container. Because of this, you should consider limiting the resources available to the container.

The following command will run panoptes_docker inside a container:

docker run -d \
    --sysctl net.core.somaxconn=511 \
    --name="panoptes_docker" \
    --shm-size=2G \
    -p 127.0.0.1:8080:3000/tcp \
    panoptes_docker

There is a five minute delay until the first metrics will show up. This is an intentional traffic calming measure.

⁠Visualize with Grafana

After launching the panoptes_docker container, you should be able to point a browser at http://127.0.0.1:8080⁠ and see the Grafana instance.

Use the auth pair admin/admin to get in, and don't worry about a new password, it won't last longer than the container. Just skip it.

The datasource and dashboard should be available immediately, however, metrics gathering is delayed by 5 minutes. Note that any changes you make to the dashboard won't survive the death of the container. If you want to learn more about Grafana, check out the docs on getting started⁠.

⁠Examine the Image

Use docker exec to access the container for examination.

docker exec -it panoptes_docker bash

If you downloaded the prebuilt image, use panoptes/panoptes_docker in this example. This will drop you into the container at /home/panoptes, and there will be some scripts available that expose the different services running in the container and split the various log files into meaningful groups.

All of these scripts are copied into the docker container from resources/misc. If you're going to interact with the python scripts in the container, you need to operate in the proper a virtual environment. To do so, run the following after you login to the container:

source /home/panoptes_v/bin/activate

conf holds the configuration scripts for Panoptes. logs holds, well, logs and run contains lock files.

⁠Useful Scripts

show-services will dump the PID and the uptime of each component.

show-influxdb.sh first dumps the data that we're interested in, then tails the logs to show the following calls. This is generally the first step in troubleshooting because InfluxDB holds the measurement data. Note: there is a 5 minute delay in the collection of stats, and the output can get a little large. To help with this, run:

/usr/bin/influx -database 'Panoptes' -format=csv -execute 'SHOW MEASUREMENTS' | more

show-logs.sh will tail all the relevant logs, but is of limited utility unless you really need a scrolling list of log updates. Then again, perhaps you run a TV crime lab.

show-panoptes.sh, show-panoptes-agents.sh, show-panoptes-discovery.sh, show-panoptes-polling.sh and show-panoptes-schedulers.sh show different aspects of the Panoptes system in operation by tailing the relevant logs.

Finally, show-snmpwalk.sh effectively queries localhost for it's SNMP OIDs. This is for checking, but also provides a glimpse into the world of OIDs.

Once you're done, stop and remove the container before trying to run another.

docker stop panoptes_docker && docker rm panoptes_docker

⁠Configuration

Panoptes is run as a python module, so the relevant code is under site-packages in the venv (/home/panoptes_v/lib/site-packages)

⁠Runtime

You can override the default localhost.json by supplying one externally and adding a flag during the container runtime. The -v join effectively overlays the default localhost.json built into the container. This example uses a localhost.json at /data/servers/panoptes/conf on the host. Use -v <source_location>:/home/panoptes/conf/localhost.json as a template.

Both -e variables are optional and default during build time to the values shown.

SNMP_SITE defaults to localhost, and is one pair of the SNMP Secrets.

SNMP_COMM_STRING defaults to public, but would be the 'community string' for the SNMP_SITE.

docker run -d \
    --sysctl net.core.somaxconn=511 \
    --name="panoptes_docker" \
    --shm-size=2G \
    -e SNMP_SITE="localhost" \
    -e SNMP_COMM_STRING="public" \
    -v /data/servers/panoptes/conf/localhost.json:/home/panoptes/conf/localhost.json \
    -p 127.0.0.1:8080:3000/tcp \
    panoptes_docker

⁠Plugins

During the build of the container, the plugins from each subdirectory of /resource/panoptes/plugins are copied into the container, so this is a good entry point into the structure.

See the Panoptes Plugin⁠ page for details

⁠Additional Resources

• Panoptes Web Site⁠
• Panoptes Source Code⁠
• Panoptes Documentation⁠
• Panoptes Concepts⁠
• Panoptes Plugin Architecture⁠
• Contributing to Panoptes⁠
• Panoptes Configuration Samples⁠
• Panoptes Plugin Examples⁠
• Open Sourcing Panoptes⁠

⁠License

This project is licensed under the terms of the Apache 2.0 open source license. Please refer to LICENSE⁠ for the full terms.

As with all Docker images, these likely also contain other software which may be under other licenses (such as Bash, etc from the base distribution, along with any direct or indirect dependencies of the primary software being contained).

As for any pre-built image usage, it is the image user's responsibility to ensure that any use of this image complies with any relevant licenses for all software contained within.