zeronorth/integration repository overview

⁠ZeroNorth Integration Container

This image can be used to integrate continuous security into your CI/CD pipeline using the ZeroNorth platform. For more information, please visit ZeroNorth at https://www.zeronorth.io⁠.

⁠Environment Variables

⁠System level environment variables

⁠Docker specific environment variables

These values must be specified if you've set STAGE to container.

⁠Specifying proxy settings

If you need to use a proxy server, you can pass the following variables in:

⁠Integration points (stages)

Integration with the ZeroNorth platform can happen in 5 separate stages:

1. Source code scanning immediately after being checked out from source control.
2. Build artifact scanning immediately after your build process.
3. Scanning of built Docker images either in a private repository or on DockerHub.
4. Shallow application scan after deployment. This is used to get a quick assessment of the integrity of newly deployed code by using non-destructive application scans.
5. Deep vulnerability scan / penn test on a clone of newly deployed infrastructure. This is used when a more comprehensive test is required, which may result in destroyed data or comprimised applications. This is only to be done against a clone of deployed infrastructure.

⁠Setting CYBRIC_ENVIRONMENT

If you're using the ZeroNorth shared SaaS platform, this should always be set to the default value of production (unless you're a ZeroNorth engineer).

If you're running your own private instance of the ZeroNorth platform in AWS, this should be set to the hostname suffix of your installation. For example, if your ZeroNorth hostname is https://fabric.example.com, then set CYBRIC_ENVIRONMENT to example.com. If your installation is at https://fabric.cybric.company.com then set CYBRIC_ENVIRONMENT to cybric.company.com

If you're running ZeroNorth in a virtual machine and the ZeroNorth services don't have the standard hostnames (api, fabric, uploads), then set CYBRIC_ENVIRONMENT to vm and set the following variables to their correct values:

• API_URL
• UI_URL
• UPLOAD_URL

For example: CYBRIC_ENVIRONMENT=vm API_URL=http://api:7123 UI_URL=http://ui:7223 UPLOAD_URL=http://upload:1080

By default the Bamboo service (API) runs on port 7123, Burlap (UI) runs on 7223 and the upload service runs on 1080. If these services are being load balanced, then just set the URLs to point at the load balancers.

⁠Including and excluding files

You can limit the number of files sent to ZeroNorth by using the INCLUDE_FILES and EXCLUDE_FILES variables. These parameters use the Unix globbing syntax.

The values passed to these variables can be a comma-separated list if you need to specify multiple types of files (eg, *.js,*.py,*.html).

If neither variable is specified, all files within the code directory will be included and sent to ZeroNorth.

If only the INCLUDE_FILES variable is specified, then only files matching the specified patterns will be included. If only the EXCLUDE_FILES variable is specified, all files with the exception of files matching the pattern will be included. If both are specified, the files in the code directory will first be filtered by the patterns in INCLUDE_FILES and then be filtered by the patterns in EXCLUDE_FILES.

If you'd like to see whether a file was included or excluded, enable debugging by setting LOG_LEVEL to debug. NOTE: This will be VERY noisy and is only recommended for debugging purposes.

⁠Retrieving a list of available scenarios

The SCENARIO variable will need to be configured with the name of the scenario that you'd like to use to scan your hosts and artifacts. To get a list of what's currently available, run the scenarios script:

Do not specify both a SCENARIO and a POLICY_ID since these are mutually exclusive.

docker run -e CYBRIC_USER=[your zeronorth username] -e CYBRIC_PASSWORD=[your zeronorth password] zeronorth/integration:latest python scenarios.py

We will return the names of the scenarios that your administrator has activated and what they can be applied to. Use this table to map a scenario's type to the STAGE that it can be run with.

⁠Volumes

The container requires 2 bind mounted volumes:

/code is the location to your source or build artifacts.

/results is the location where the container will dump the scan results to a file named scan.out

NOTE: If you're using S3_PATH, then you don't need to mount the /code volume since your code will be pulled from S3.

You can optionally add a third volume mount to use a custom certificate authority. This is useful if you're running a private ZeroNorth instance and you've signed your SSL certificates with your own certificate authority. This will enable the integration container to trust your CA.

Create a folder on your local or build machine containing your CA root certificate in pem format and mount that directory to /usr/local/share/ca-certificates

For example:

-v /mnt/custom_ca:/usr/local/share/ca-certificates/

On startup, the integration container will copy any certificates in /mnt/custom_ca to the trust store so that HTTPS calls to the ZeroNorth API will validate properly.

⁠Mounting a CUSTOMER_ARTIFACTS_PATH

The CUSTOMER_ARTIFACTS_PATH should reference a customer_artifacts_folder which contains artifacts that will be presented to the runner. The folder structure described bellow is important for the runner to see those artifact(s).

Adding a customer provided certificate (PEM format). The certificate needs to be named certificate.

customer_artifacts_folder/certificates/certificate

Adding a license which is required for a specific tool scan. License name specified during license creation on the ZeroNorth platform.

customer_artifacts_folder/license/<License Name>

Adding ZAP context files (auth script, context, url file, etc). The artifact name specified during artifact creation on the ZeroNorth platform.

customer_artifacts_folder/authScript/<Artifact Name>
customer_artifacts_folder/context/<Artifact Name>
customer_artifacts_folder/urlFile/<Artifact Name>

Adding a ZAP hosts file. The hosts file name is specified during creation of the host file on the ZeroNorth platform.

customer_artifacts_folder/hosts/<Hosts file name>

⁠Scanning source code

Performing a source code scan is done by setting the STAGE environment variable to SOURCE and calling the integration container:

docker run -v [host location of checked out source code]:/code -v [host location of results]:/results -e LOG_LEVEL=DEBUG -e CYBRIC_USER=[your zeronorth username] -e CYBRIC_PASSWORD=[your zeronorth password] -e SCENARIO=brakeman-default -e STAGE=SOURCE zeronorth/integration:latest python cybric.py

For example, if I had just completed a step to check out source code from another container, and that container had written the source code to a bind-mounted volume on the Docker host called /mnt/shared/source, my source code scan would be executed like this:

docker run -v /mnt/shared/source:/code -v /mnt/shared/results:/results -e LOG_LEVEL=DEBUG -e [email protected] -e CYBRIC_PASSWORD=****** -e SCENARIO=brakeman-default -e STAGE=SOURCE zeronorth/integration:latest python cybric.py

The source code will be compressed and sent to ZeroNorth for scanning and analysis. The results of this scan will be stored on the ZeroNorth platform, ingested for analytic analysis and results returned for consumption by your system. Results will be returned as a JSON structure for easy programmatic parsing.

If you host your source and build artifacts in S3, you can pull them down by referencing them with the S3_PATH environment variable. If you need to specify your AWS IAM credentials, you can also pass in AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

If your code lives in a subdirectory on S3, scan it like this:

docker run -it -e AWS_ACCESS_KEY_ID=your-access-key-here -e AWS_SECRET_ACCESS_KEY=your-secret-key-here -e SCENARIO=sonarqube-agent -e [email protected] -e CYBRIC_PASSWORD=****** -e STAGE=source -e S3_PATH=s3://your-bucket-name/directory-name/ zeronorth/integration:latest python cybric.py

You can also reference .zip and .tgz/.tar.gz files in S3 and they will be pulled down. For .tgz/.tar.gz files, they are sent directly to ZeroNorth for scanning, but .zip files need to be downloaded and recompressed as .tgz files.

NOTE Make sure that your archive has all of the code that you want to scan under a single top-level directory.

If you set FAIL_ON_HIGH_SEVERITY to 1 then any high severity issues found during scan will cause the container to exit with a code of 1. This can be helpful where the ZeroNorth integration container is used in a build pipeline and you'd like the pipeline to fail when high severity errors are found.

⁠Scanning hosts for vulnerabilities

In addition to scanning source, build and container artifacts for vulnerabilities, ZeroNorth can also scan deployed applications for vulnerabilities as well.

To do an nmap discovery scan of a server

docker run -v /mnt/shared/results:/results -e LOG_LEVEL=DEBUG -e CYBRIC_USER=[your zeronorth username] -e CYBRIC_PASSWORD=[your zeronorth password] -e SCENARIO=nmap-discovery -e STAGE=HOST HOST_TARGET=[ip or host of the server to scan] zeronorth/integration:latest python cybric.py

NOTE Be extremely careful about using some of the more invasive tools and configurations against important hosts. Many of these tools have aggressive settings that will actively try to compromise any exploits that they find, which can be very useful in penetration testing.

⁠ZeroNorth Applications

If you have an Application in your ZeroNorth account and you'd like to ensure that none of the Targets within them have vulnerabilities above a criticality threshold, you can call the checkapp.py script.

Go to FabricOPS > Application, locate your appication, and then grab the application's ID by clicking the icon next to the application's name. Run the checkapp.py script as follows:

docker run -e APPLICATION_ID={your applications ID} -e CYBRIC_API_KEY={your API key} FAIL_ON_HIGH_SEVERITY={severity level to fail on} zeronorth/integration:latest python checkapp.py

The severity level can be one of INFO, LOW, MEDIUM, HIGH or CRITICAL. If any of the Targets in the specified Application have vulnerabilities at or exceeding the specified criticality level, the command will exit with a status code of 1. This will allow you to fail builds based on vulnerabilities across many targets and ensure that the entire Application is secure.

⁠Cybricfile

Instead of specifying the scan parameters using environment variables, you can also use a Cybricfile. A Cybricfile is a YAML file checked in at the base of your project's code tree named .cybric.yml.

version: '1.0'
output_format: json
log_level: DEBUG
source:
  project_name: Project A Source
  fail_on_high_severity: true
  include_files:
    - src/**
  exclude_files:
    - build/**
  scenario: brakeman-default

build:
  project_name: Project A Build

container:
  project_name: Project A Container
  scenario: docker-image-scan-default
  image: mycompany/projecta:latest

host:
  project_name: Project A Web Server
  scenario: nmap-discovery
  host_target: staging.mycompany.com

The top-level keys correlate to the STAGE. To execute the source stage with this file checked into the project tree, execute the command like this:

docker run -v [host location of checked out source code]:/code -v [host location of results]:/results -e LOG_LEVEL=DEBUG -e CYBRIC_USER=[your zeronorth username] -e CYBRIC_PASSWORD=[your zeronorth password] -e STAGE=SOURCE zeronorth/integration:latest python cybric.py

NOTE: Environment variables passed in will override any values specified in the Cybricfile.

NOTE: You can not specify keys or secrets in the Cybricfile. This was done intentionally to prevent secrets from being checked into version control. All keys & secrets must be passed in as environment variables.

⁠Retrieving scan results

The results will be written to /results/scan.out in the format specified in OUTPUT_FORMAT.

⁠Windows EXE Creation

Windows 10 has some... complexities with pyinstaller

⁠PreInstall Steps

Install python3.6.2

https://www.python.org/ftp/python/3.6.2/python-3.6.2-amd64.exe⁠

Install Windows 10 SDK

https://go.microsoft.com/fwlink/?linkid=859886⁠

Set env path for python, python scripts and windows 10 sdk x64 dlls

C:\Users\Administrator\AppData\Local\Programs\Python\Python36
C:\Users\Administrator\AppData\Local\Programs\Python\Python36\Scripts
C:\Program Files (x86)\Windows Kits\10\Redist\ucrt\Dlls\x64

Install virtualenv

pip install virtualenv==16.1

Activate a new virtualenv in the integration code directory

cd C:\path\to\integration
virtualenv venv
. venv\Scripts\activate

Install pycurl wheel since it has dependency issues on Windows otherwise

pip install lib\pycurl-7.43.0-cp36-cp36m-win_amd64.whl

Install dependencies

pip install -r requirements.txt
python setup.py install
pip install pywin32

Install pyinstaller

pip install pyinstaller

Run pyinstaller to create a single exe binary to run on windows 10 based systems

pyinstaller --onefile --additional-hooks-dir=.\hooks --distpath=.\bin\windows_amd64 cybric_cli.py --path "C:\Program Files (x86)\Windows Kits\10\Redist\ucrt\Dlls\x64"

⁠Linux CLI Creation

Before you can create the CLI, you will need credentials to the ZeroNorth Nexus server. Export your Nexus username and password before running the following make task:

export NEXUS_USERNAME={your nexus username}
export NEXUS_PASSWORD={your nexus password}

Once this is done, you can now run make centos_cli for a CentOS/RHEL version or make ubuntu_cli for an Ubuntu/Debian version.

When it finishes, you will have a file called cybric_cli in the appropriate folder in bin (eg. bin/centos7_amd64 or bin/ubuntu_amd64). It does use Docker to build the CLI, so you will need to have the Docker daemon running.

Copyright © 2017, 2018, 2019 ZeroNorth, Inc. All rights reserved.