CTF Jeopardy Headless

This container provides a unique headless browser which can be used by many different CTF challenges requiring a headless browser bot, such as the one needed for XSS, CSRF, etc.

The container exposes an HTTP API which can be used to instruct workers controlling a remote instance of Firefox or Chrome. The web server provides basic authentication via an auth token to prevent DoS, and avoids common issues such as zombie processes and hanging browsers. The browser is not running in incognito but everything is still cleaned up after each job.

Getting started

The system needs at least three containers: a manager, a worker, and a RabbitMQ instance used to communicate between the two and schedule jobs. Workers can be scaled in order to have multiple concurrent browser sessions.

The management container exposes a web server listening on port 5000. The only required configuration is the secret auth token which will be used by legitimate clients to authenticate to the server (this is a measure to avoid DoS).

The headless can be run with docker compose:

headless:
  image: cybersecnatlab/challenge-headless:latest-manager
  restart: unless-stopped
  environment:
    AUTH_TOKEN: supersecret
    RABBITMQ_HOST: headless-rabbitmq
    RABBITMQ_QUEUE: headless-jobs
  depends_on:
    - headless-rabbitmq

headless-rabbitmq:
  image: rabbitmq:3.11
  restart: unless-stopped

headless-worker:
  image: cybersecnatlab/challenge-headless:latest-worker
  restart: unless-stopped
  environment:
    RABBITMQ_HOST: headless-rabbitmq
    RABBITMQ_QUEUE: headless-jobs
  deploy:
    replicas: 4
  depends_on:
    - headless-rabbitmq

The current version of the workers always keeps a browser ready to go. This might use more resources than needed, if you don't really need a browser ready right now and can wait for it to start up, you can set the environment variable LEGACY_MODE to true in the worker container(s).

Usage

The API exposes two endpoints, both are authenticated using the X-Auth header containing the authorization token for the headless service:

• POST / allows to create a new job, and it expects a JSON body containing the following values:

  key	required	type	description
  actions	:heavy_check_mark:	array	Describes the list of actions to be performed by the headless browser
  timeout	:x:	number	Defines the time in seconds after which the worker will be terminated
  browser	:x:	'firefox' | 'chrome'	Specifies what browser to use. DEFAULTS TO FIREFOX. See browser support for more information.

  When a request is received, it is put in a queue for workers to execute. The worker opens a new browser instance, executes all the requested actions in order and then fully closes the browser.

• GET /jobs/<job_id> allows to get the job status in JSON format:

  key	type	description
  status	'queued' | 'running' | 'finished'	Denotes the current status of the job
  queued_at	number	When the job was queued in UNIX seconds
  running_at	number | null	When the job started running on a worker in UNIX seconds
  finished_at	number | null	When the job was finished in UNIX seconds

Available actions

Every action is itself a JSON object and contains a type value, which is used to define what kind of action will be executed. The available action types are:

• request
• sleep
• set-cookie
• delete-cookie
• delete-all-cookies
• click
• type

Here is an example on how a request could be made to the headless service using curl:

curl -X POST -H 'Content-Type: application/json' -H 'X-Auth: supersecret' --data '{"actions": [{"type": "request", "url": "http://example.com"}]}' http://localhost:5000

What follows is the documentation for each action type:

request

This is used to make the browser request a specific page. Note that the headless browser is set up so that we can also do requests with methods other than GET, so we can use this action type to submit forms etc. If that's not enough, you can use the click and type actions

Here are the fields for this action:

Examples:

{
  "type": "request",
  "url": "http://example.com",
  "timeout": 5
}

{
  "type": "request",
  "url": "http://example.com/login",
  "method": "POST",
  "data": "username=admin&password=admin",
  "headers": {
    "Content-Type": "application/x-www-form-urlencoded"
  }
}

sleep

This is used to just... wait for a little bit. Can be useful to allow for some JavaScript to execute.

Here are the fields for this action:

Examples:

{
  "type": "sleep",
  "time": 5
}

{
  "type": "sleep",
  "time": 3.1415
}

set-cookie

This is used to set a cookie for a specific domain. IMPORTANT: you cannot specify a domain from this action. If you want to set a cookie for http://example.com⁠ you have to first visit http://example.com⁠ (with the request action) and then use this action to set the cookie.

Here are the fields for this action:

Examples:

{
  "type": "set-cookie",
  "name": "session",
  "value": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwidXJsIjoiaHR0cHM6Ly93d3cueW91dHViZS5jb20vd2F0Y2g_dj1kUXc0dzlXZ1hjUSIsImlhdCI6MTUxNjIzOTAyMn0",
  "httpOnly": true
}

delete-cookie

This is used to delete a cookie for a specific domain. IMPORTANT: you cannot specify a domain from this action. If you want to delete a cookie for http://example.com⁠ you have to first visit http://example.com⁠ (with the request action) and then use this action to delete the cookie.

Here are the fields for this action:

Examples:

{
  "type": "delete-cookie",
  "name": "session"
}

delete-all-cookies

This is used to delete every cookie set in the browser.

Here are the fields for this action:

Examples:

{
  "type": "delete-all-cookies"
}

click

This is used to click on a DOM element. The element is selected using a CSS selector (such as #login or .btn[href="/login"]). The browser automatically waits for it to be present in the document and scrolls the page to get it into the viewport. When possible, using directly the request action is the preferred and faster solution for submitting forms, use this only when DOM interaction is strictly required.

Here are the fields for this action:

Examples:

{
  "type": "click",
  "element": "button#submit"
}

type

This is used to type an input into a DOM element. The element is selected using a CSS selector (such as #login or .btn[href="/login"]). The browser automatically waits for it to be present in the document and scrolls the page to get it into the viewport. The browser has a 100ms delay between each key press. When possible, using directly the request action is the preferred and faster solution for submitting forms, use this only when DOM interaction is strictly required.

Here are the fields for this action:

Examples:

{
  "type": "type",
  "element": "#password",
  "value": "supersecretpassword"
}

Debugging

Each request to the server returns a JSON containing a job id. This job id is added to each log generated by the worker. All logs are written to the stdout and stderr of the container.

Logs contains a list every action performed by the browser and a list of all the URLs which have been requested during the job execution (it's nice to see if a request to ngrok/requestbin has been actually sent).

These logs are extremely useful during local challenge development, trust me.

Browser support

This headless supports both Firefox and Google Chrome. Since the original version of the headless used Firefox, for backwards compatibility Firefox is still the default option.

The feature sets of the two browsers are equivalent, with a small but potentially important difference: Firefox is configured to skip the security warning presented when submitting data from HTTPS to HTTP. The warning cannot be disabled in Chrome. This can be relevant if you serve your web challenges in HTTP and the player is attempting an attack (something like CSRF) from HTTPS (as it's the case with the new default ngrok tunnels). Such an attack would work on Firefox but not on Chrome. While ultimately this is a player issue, you might get a lot of support tickets because " the headless is not working and the challenge is unsolvable". Keep this in mind. Maybe use HTTPS for your challenges if you intend to use Chrome and you fear this might become an issue.

If you want the nitty-gritty details, here are the specific browser versions running on the headless:

Configuration

Other than the mandatory AUTH_TOKEN variable, there are additional optional configuration options which can be specified in the environment, although the default values are already quite sensible:

• JOBS_EXPIRATION (default 30): jobs that have been queued for more than the specified number of minutes will be discarded to avoid clogging the queue
• JOBS_STATUS_TTL (default 30): job statuses will be kept in memory for the specified number of minutes since they were queued and then discarded
• WORKER_TIMEOUT (default 30): if no worker timeout is specifically provided in the HTTP request, this is the maximum time in seconds the worker is allowed to execute before being killed. Since each action performed by the worker has its own timeout, the worker timeout might actually be less than this value.
• WORKER_TIMEOUT_HARD_LIMIT (default 120): this is the absolute maximum time in seconds a worker is allowed to run before getting killed. This puts a cap both to the WORKER_TIMEOUT value and to the custom timeout passed in the HTTP requests.
• DEFAULT_REQUEST_TIMEOUT (default 10): This is the default timeout in seconds for a browser request. If the request takes more than this limit the worker exits.
• WORKERS_COUNT (default 4): This is the number of concurrent workers running inside the container
• FIND_ELEMENT_TIMEOUT (default 5): This is the maximum time the browser waits for an element to enter the DOM when using the actions click and type