cybersecnatlab/challenge-headless

By cybersecnatlab

Updated 5 months ago

Headless browser as a Service

Image
Message Queues
Developer Tools
0

3.1K

CTF Jeopardy Headless

This container provides a unique headless browser which can be used by many different CTF challenges requiring a headless browser bot, such as the one needed for XSS, CSRF, etc.

The container exposes an HTTP API which can be used to instruct workers controlling a remote instance of Firefox or Chrome. The web server provides basic authentication via an auth token to prevent DoS, and avoids common issues such as zombie processes and hanging browsers. The browser is not running in incognito but everything is still cleaned up after each job.

Getting started

The system needs at least three containers: a manager, a worker, and a RabbitMQ instance used to communicate between the two and schedule jobs. Workers can be scaled in order to have multiple concurrent browser sessions.

The management container exposes a web server listening on port 5000. The only required configuration is the secret auth token which will be used by legitimate clients to authenticate to the server (this is a measure to avoid DoS).

The headless can be run with docker compose:

headless:
  image: cybersecnatlab/challenge-headless:latest-manager
  restart: unless-stopped
  environment:
    AUTH_TOKEN: supersecret
    RABBITMQ_HOST: headless-rabbitmq
    RABBITMQ_QUEUE: headless-jobs
  depends_on:
    - headless-rabbitmq

headless-rabbitmq:
  image: rabbitmq:3.11
  restart: unless-stopped

headless-worker:
  image: cybersecnatlab/challenge-headless:latest-worker
  restart: unless-stopped
  environment:
    RABBITMQ_HOST: headless-rabbitmq
    RABBITMQ_QUEUE: headless-jobs
  deploy:
    replicas: 4
  depends_on:
    - headless-rabbitmq

The current version of the workers always keeps a browser ready to go. This might use more resources than needed, if you don't really need a browser ready right now and can wait for it to start up, you can set the environment variable LEGACY_MODE to true in the worker container(s).

Usage

The API exposes two endpoints, both are authenticated using the X-Auth header containing the authorization token for the headless service:

  • POST / allows to create a new job, and it expects a JSON body containing the following values:

    keyrequiredtypedescription
    actions:heavy_check_mark:arrayDescribes the list of actions to be performed by the headless browser
    timeout:x:numberDefines the time in seconds after which the worker will be terminated
    browser:x:'firefox' | 'chrome'Specifies what browser to use. DEFAULTS TO FIREFOX. See browser support for more information.

    When a request is received, it is put in a queue for workers to execute. The worker opens a new browser instance, executes all the requested actions in order and then fully closes the browser.

  • GET /jobs/<job_id> allows to get the job status in JSON format:

    keytypedescription
    status'queued' | 'running' | 'finished'Denotes the current status of the job
    queued_atnumberWhen the job was queued in UNIX seconds
    running_atnumber | nullWhen the job started running on a worker in UNIX seconds
    finished_atnumber | nullWhen the job was finished in UNIX seconds

Available actions

Every action is itself a JSON object and contains a type value, which is used to define what kind of action will be executed. The available action types are:

Here is an example on how a request could be made to the headless service using curl:

curl -X POST -H 'Content-Type: application/json' -H 'X-Auth: supersecret' --data '{"actions": [{"type": "request", "url": "http://example.com"}]}' http://localhost:5000

What follows is the documentation for each action type:

request

This is used to make the browser request a specific page. Note that the headless browser is set up so that we can also do requests with methods other than GET, so we can use this action type to submit forms etc. If that's not enough, you can use the click and type actions

Here are the fields for this action:

keyrequiredtypedefault valuedescription
type:heavy_check_mark:"request"Specifies the action type
url:heavy_check_mark:stringSpecifies the URL to request
method:x:string"GET"Specifies the HTTP method used to perform the request
data:x:stringDefines the request body. Both the encoding and the Content-Type are responsibility of the caller. The Content-Type must be provided in the headers field. The Content-Length will be computed by the caller, so there's no need to specify it
headers:x:dict{}Specifies the request headers to be send with the request
timeout:x:numberDEFAULT_REQUEST_TIMEOUTSpecifies the timeout in seconds for the request to execute. If the request fails to load in time or the browser hangs or something else happens preventing the request from finishing, the worker is killed

Examples:

{
  "type": "request",
  "url": "http://example.com",
  "timeout": 5
}
{
  "type": "request",
  "url": "http://example.com/login",
  "method": "POST",
  "data": "username=admin&password=admin",
  "headers": {
    "Content-Type": "application/x-www-form-urlencoded"
  }
}
sleep

This is used to just... wait for a little bit. Can be useful to allow for some JavaScript to execute.

Here are the fields for this action:

keyrequiredtypedefault valuedescription
type:heavy_check_mark:"sleep"Specifies the action type
time:heavy_check_mark:numberThe time in seconds the worker will wait

Examples:

{
  "type": "sleep",
  "time": 5
}
{
  "type": "sleep",
  "time": 3.1415
}
set-cookie

This is used to set a cookie for a specific domain. IMPORTANT: you cannot specify a domain from this action. If you want to set a cookie for http://example.com you have to first visit http://example.com (with the request action) and then use this action to set the cookie.

Here are the fields for this action:

keyrequiredtypedefault valuedescription
type:heavy_check_mark:"set-cookie"Specifies the action type
name:heavy_check_mark:stringSpecifies the name of the cookie to set
value:heavy_check_mark:stringSpecifies the value of the cookie
httpOnly:x:booleanLeft to the browserSpecifies whether the cookie has to be set "Http Only"
sameSite:x:"Lax" | "Strict" | "None"Left to the browserSpecifies the Same Site configuration for the cookie

Examples:

{
  "type": "set-cookie",
  "name": "session",
  "value": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwidXJsIjoiaHR0cHM6Ly93d3cueW91dHViZS5jb20vd2F0Y2g_dj1kUXc0dzlXZ1hjUSIsImlhdCI6MTUxNjIzOTAyMn0",
  "httpOnly": true
}
delete-cookie

This is used to delete a cookie for a specific domain. IMPORTANT: you cannot specify a domain from this action. If you want to delete a cookie for http://example.com you have to first visit http://example.com (with the request action) and then use this action to delete the cookie.

Here are the fields for this action:

keyrequiredtypedefault valuedescription
type:heavy_check_mark:"delete-cookie"Specifies the action type
name:heavy_check_mark:stringSpecifies the name of the cookie to delete

Examples:

{
  "type": "delete-cookie",
  "name": "session"
}
delete-all-cookies

This is used to delete every cookie set in the browser.

Here are the fields for this action:

keyrequiredtypedefault valuedescription
type:heavy_check_mark:"delete-all-cookies"Specifies the action type

Examples:

{
  "type": "delete-all-cookies"
}
click

This is used to click on a DOM element. The element is selected using a CSS selector (such as #login or .btn[href="/login"]). The browser automatically waits for it to be present in the document and scrolls the page to get it into the viewport. When possible, using directly the request action is the preferred and faster solution for submitting forms, use this only when DOM interaction is strictly required.

Here are the fields for this action:

keyrequiredtypedefault valuedescription
type:heavy_check_mark:"click"Specifies the action type
element:heavy_check_mark:stringIndicates the CSS selector used to find the element to click

Examples:

{
  "type": "click",
  "element": "button#submit"
}
type

This is used to type an input into a DOM element. The element is selected using a CSS selector (such as #login or .btn[href="/login"]). The browser automatically waits for it to be present in the document and scrolls the page to get it into the viewport. The browser has a 100ms delay between each key press. When possible, using directly the request action is the preferred and faster solution for submitting forms, use this only when DOM interaction is strictly required.

Here are the fields for this action:

keyrequiredtypedefault valuedescription
type:heavy_check_mark:"type"Specifies the action type
element:heavy_check_mark:stringIndicates the CSS selector used to find the element to click
value:heavy_check_mark:stringIndicates the input to type into the element
delay:x:number0.1Specifies the delay time in seconds between each keystroke. Might be used to simulate a human or to allow for event handlers to execute properly

Examples:

{
  "type": "type",
  "element": "#password",
  "value": "supersecretpassword"
}

Debugging

Each request to the server returns a JSON containing a job id. This job id is added to each log generated by the worker. All logs are written to the stdout and stderr of the container.

Logs contains a list every action performed by the browser and a list of all the URLs which have been requested during the job execution (it's nice to see if a request to ngrok/requestbin has been actually sent).

These logs are extremely useful during local challenge development, trust me.

Browser support

This headless supports both Firefox and Google Chrome. Since the original version of the headless used Firefox, for backwards compatibility Firefox is still the default option.

The feature sets of the two browsers are equivalent, with a small but potentially important difference: Firefox is configured to skip the security warning presented when submitting data from HTTPS to HTTP. The warning cannot be disabled in Chrome. This can be relevant if you serve your web challenges in HTTP and the player is attempting an attack (something like CSRF) from HTTPS (as it's the case with the new default ngrok tunnels). Such an attack would work on Firefox but not on Chrome. While ultimately this is a player issue, you might get a lot of support tickets because " the headless is not working and the challenge is unsolvable". Keep this in mind. Maybe use HTTPS for your challenges if you intend to use Chrome and you fear this might become an issue.

If you want the nitty-gritty details, here are the specific browser versions running on the headless:

browser/driverversion
firefox119.0
geckodriver0.34.0
chrome-for-testing122.0.6248.0
chromedriver-for-testing122.0.6248.0

Configuration

Other than the mandatory AUTH_TOKEN variable, there are additional optional configuration options which can be specified in the environment, although the default values are already quite sensible:

  • JOBS_EXPIRATION (default 30): jobs that have been queued for more than the specified number of minutes will be discarded to avoid clogging the queue
  • JOBS_STATUS_TTL (default 30): job statuses will be kept in memory for the specified number of minutes since they were queued and then discarded
  • WORKER_TIMEOUT (default 30): if no worker timeout is specifically provided in the HTTP request, this is the maximum time in seconds the worker is allowed to execute before being killed. Since each action performed by the worker has its own timeout, the worker timeout might actually be less than this value.
  • WORKER_TIMEOUT_HARD_LIMIT (default 120): this is the absolute maximum time in seconds a worker is allowed to run before getting killed. This puts a cap both to the WORKER_TIMEOUT value and to the custom timeout passed in the HTTP requests.
  • DEFAULT_REQUEST_TIMEOUT (default 10): This is the default timeout in seconds for a browser request. If the request takes more than this limit the worker exits.
  • WORKERS_COUNT (default 4): This is the number of concurrent workers running inside the container
  • FIND_ELEMENT_TIMEOUT (default 5): This is the maximum time the browser waits for an element to enter the DOM when using the actions click and type

Docker Pull Command

docker pull cybersecnatlab/challenge-headless