doublevkay/codeql-agent-dev

•Updated almost 4 years ago

Create database and execute CodeQL analysis

Image

2.2K

doublevkay/codeql-agent-dev repository overview

⁠CodeQL Agent for Docker

CodeQL Agent⁠ is a project aimed at automating the use of CodeQL. The project helps create database and execute CodeQL analysis. CodeQL Agent is a Docker image. It is designed to be compatible with SAST Gitlab CI/CD⁠.

CodeQL Agent for Docker is also the base image of CodeQL Agent for Visual Studio Code⁠ - an extension for Visual Studio Code⁠ that simplifies CodeQL usage and executes code scanning automatically.

The CodeQL Agent image is released on Docker Hub under the name doublevkay/codeql-agent-dev⁠. You can use it without building locally.

⁠What is this for?

CodeQL Agent for Docker provides these key features:

Detecting language automatically.
Creating CodeQL database.
Executing CodeQL database analysis.
Integrating CodeQL to your CI/CD process.
Integrating CodeQL code scanning to Gitlab CI/CD⁠.

⁠Getting Started

Bind mounts⁠ the source, the results folder and run codeql-agent-dev image with the following docker command.

docker run --rm --name codeql-agent-docker \
  -v "$PWD:/opt/src" \
  -v "$PWD/codeql-agent-results:/opt/results" \
  doublevkay/codeql-agent-dev

You also can specify more options to run CodeQL Agent. See Supported options⁠ for more details.

⁠Supported options

You can set environment variables to use the following supported options:

Variable	Description
`LANGUAGE`	Value `<language>`. Set project language to build database or execute SAST. The `<language>` must be: `python`, `javascript`, `cpp`, `csharp`, `java`, `go`, `typescript`, `c`.
`USERID`	Value `<id>`. Set the owner of the results folder to `<id>`.
`GROUPID`	Value `<group_id>`. Set the group owner of the results folder to `<group_id>`.
`THREADS`	Value `<number_of_threads>`. Use this many threads to build database and evaluate queries. Defaults to 1. You can pass 0 to use one thread per core on the machine.
`OVERWRITE_FLAG`	Value `--overwrite`. Enable/disable overwrite database when database path exists and not an empty directory. This flag is useful for forcibly rebuilding the database.
`QS`	Value `<queries-suite>`. Specify a list of queries to run over your database. The default value is `<language>-security-extended.qls`. For more details, please see Analyzing databases with the CodeQL CLI⁠.
`SAVE_CACHE_FLAG`	Value `--save-cache`. Aggressively save intermediate results to the disk cache. This may speed up subsequent queries if they are similar. Be aware that using this option will greatly increase disk usage and initial evaluation time.
`ACTION`	Value `create-database-only`. Creating CodeQL database only without executing CodeQL analysis.
`COMMAND`	Value `<command>`. The variable used when you create a CodeQL database for one or more compiled languages, omit if the only languages requested are Python and JavaScript. This specifies the build commands needed to invoke the compiler. If you don't set this variable, CodeQL will attempt to detect the build system automatically, using a built-in autobuilder.
`JAVA_VERSION`	Value `<java_version>`. Set the Java version. The default Java version is Java 11. It must be `8` or `11`.

Disclaimer: CodeQL Agent directly forwards these options to the command arguments while running the container. Please take it as your security responsibility.

⁠Examples usage

Basic code scanning.

docker run --rm --name codeql-agent-docker \
  -v "$PWD:/opt/src" \
  -v "$PWD/codeql-agent-results:/opt/results" \
  doublevkay/codeql-agent-dev

Code scanning with maximum threads available.

docker run --rm --name codeql-agent-docker \
  -v "$PWD:/opt/src" \
  -v "$PWD/codeql-agent-results:/opt/results" \
  -e "THREADS=0" \
  doublevkay/codeql-agent-dev

Create database only.

docker run --rm --name codeql-agent-docker \
  -v "$PWD:/opt/src" \
  -v "$PWD/codeql-agent-results:/opt/results" \
  -e "ACTION=create-database-only" \
  doublevkay/codeql-agent-dev

Specify the queries suite for Java source.

docker run --rm --name codeql-agent-docker \
  -v "$PWD:/opt/src" \
  -v "$PWD/codeql-agent-results:/opt/results" \
  -e "LANGUAGE=java" \
  -e "QS=java-security-and-quality.qls" \
  doublevkay/codeql-agent-dev

Change owner of the results folder.

Because CodeQL Agent runs the script as root in Docker containers. So maybe you need to change the results folder owner to your own.

docker run --rm --name codeql-agent-docker \
  -v "$PWD:/opt/src" \
  -v "$PWD/codeql-agent-results:/opt/results" \
  -e "USERID=$(id -u ${USER})" -e "GROUPID=$(id -g ${USER}) \
  doublevkay/codeql-agent-dev

Specify the Java version and the build database command

docker run --rm --name codeql-agent-docker \
  -v "$PWD:/opt/src" \
  -v "$PWD/codeql-agent-results:/opt/results" \
  -e "LANGUAGE=java" \
  -e "JAVA_VERSION=8" \
  -e "COMMAND=mvn clean install" \
  doublevkay/codeql-agent-dev

⁠Integrate CodeQL into GitLab CI/CD

See demo⁠

You can integrate CodeQL into Gitlab CI/CD by setting up the .gitlab-ci.yml file with the following template:

codeql:
  image: doublevkay/codeql-agent-dev
  script: /root/scripts/analyze.sh
  artifacts:
    reports:
      sast: gl-sast-report.json

You can use supported options⁠ by setting environment variables (see GitLab CI/CD variables⁠). For example, the following setup will use four threads to execute static application security testing in Gitlab CI/CD for Java language.

codeql:
  image: doublevkay/codeql-agent-dev
  script: /root/scripts/analyze.sh
  artifacts:
    reports:
      sast: gl-sast-report.json
  variables:
    LANGUAGE: java
    THREADS: 4

⁠Build

You can use CodeQL Agent Image⁠ on Docker Hub or customize and build it locally⁠.

# Build codeql-agent-dev docker image locally 
cd codeql-agent
docker build -t codeql-agent-dev .

⁠How does it work?

CodeQL Agent is a Docker image. The following steps are done to achieve the goals of automating the use of CodeQL and integrating it to Gitlab CI/CD.

Setting up environment

In this step, the image prepares the environment for executing CodeQL. It includes: using Ubuntu base image; downloading and installing CodeQL Bundle⁠ (which contains the CodeQL CLI and the precompiled library queries to reduce the CodeQL execution time); installing necessary softwares such as java, maven, nodejs, typescript,... to create a CodeQL database successfully.

Detecting language

CodeQL Agent uses github/linguist⁠ to detect the source code language.

Creating database

CodeQL Agent runs the CodeQL create database command.

codeql database create --threads=$THREADS --language=$LANGUAGE $COMMAND $DB -s $SRC $OVERWRITE_FLAG

Specifying query suites

Analyzing databases requires specifying a query suite. According to the goals of application static application security testing (SAST) goals, CodeQL Agent uses <language>-security-extended.qls as the default query suite.

Analyzing database

CodeQL Agent runs the CodeQL database analysis command.

codeql database analyze --format=$FORMAT --threads=$THREADS $SAVE_CACHE_FLAG --output=$OUTPUT/issues.$FORMAT $DB $QS

Converting result format

Gitlab CI/CD⁠ does not support the SARIF format. Therefore, CodeQL Agent will convert the CodeQL result from SARIF format⁠ to Security Report Schemas⁠ (provided by Gitlab). This step is done by mapping the fields of two formats. The details of implementation are in the sarif2sast⁠ script. You can use this script independently as a workaround to solve the Gitlab Issue 118496⁠.