threathuntproj/hunting repository overview

Build instructions for this image are available at https://github.com/ThreatHuntingProject/hunter⁠.

⁠Overview

This image contains a complete threat hunting & data analysis environment built on Python, Pandas, PySpark and Jupyter notebook. It is provided in the hopes that it makes getting started with analytic style hunting a bit easier by providing a turnkey platform that "just works".

⁠Downloading the image

You can pull any of these images just as you would with any other image, using docker pull. For example, to get the most recent stable image:

docker pull threathuntproj/hunting:latest 

You can also pull a specific version of the image using it's date, like so:

docker pull threathuntproj/hunting:20180831 

These images are built automatically via the CI system. Each time we build, an image is created and pushed that contains both the date code and the build number (e.g., threathuntproj/hunting:20180831.37). In general, you can ignore the images tagged with build numbers. The "date code only" images are always the the most up-to-date from that date.

⁠What's in the Image?

This image is built on the official Jupyter PySpark image, with some additional packages that will be useful for hunting.

The following is a partial list of the major features:

• Support for either the traditional Notebook or the new Lab interface
• Built-in extensions manager for the Notebook interface
• Python 3 (default) and Python 2 kernels

In addition, there are several preinstalled libraries useful for data search, analysis and visualization, including:

• elasticsearch-dsl
• splunk-sdk
• pandas
• pyspark
• numpy
• matplotlib
• seaborn
• plotly (with cufflinks support)
• scikit-learn
• ipywidgets

⁠Running the Image

To use the traditional notebook interface, run the container like so:

docker run -it -p 8888:8888 -e GEN_CERT=yes \
-v $WHERE_YOU_WANT_NOTEBOOKS_TO_LIVE:/home/jovyan/work \
threathuntproj/hunting:latest

If you'd like to use the new Jupyter Lab environment, just set the JUPYTER_ENABLE_LAB variable at runtime:

docker run -it -p 8888:8888 -e GEN_CERT=yes \
-e JUPYTER_ENABLE_LAB=yes \ 
-v $WHERE_YOU_WANT_NOTEBOOKS_TO_LIVE:/home/jovyan/work \ 
threathuntproj/hunting:latest

$WHERE_YOU_WANT_NOTEBOOKS_TO_LIVE can be any directory on your system. Some may wish to set it to $HOME so that all their files will be available in the notebook environment. Others may prefer to use a specific subdirectory, to avoid exactly that type of file sharing. The choice is up to you.

As a special feature, the $WHERE_YOU_WANT_NOTEBOOKS_TO_LIVE/lib directory is part of the PYTHONPATH in the container. If you install a python module into that directory, your notebooks will automatically be able to find it when they're running in the container. This provides a convenient way for you to add your own modules without having to rebuild the entire image.

⁠Accessing the Notebook Environment

When the notebook server runs, it will print the UI URL to the console. This URL contains a randomly-generated access token, which will serve instead of a password to authenticate you to the notebook server. Click that URL, or cut-n-paste it into your browser, and you'll be logged in.

The image requires TLS in order to access the notebooks, so beware if you use this with Kitematic or similar UI. Even though they will direct you to the correct port, Kitematic creates an HTTP URL, which is incorrect. Simply change it to use HTTPS and you'll be able to access the UI.