cybersecnatlab/challenge-headless
Headless browser as a Service
3.1K
This container provides a unique headless browser which can be used by many different CTF challenges requiring a headless browser bot, such as the one needed for XSS, CSRF, etc.
The container exposes an HTTP API which can be used to instruct workers controlling a remote instance of Firefox or Chrome. The web server provides basic authentication via an auth token to prevent DoS, and avoids common issues such as zombie processes and hanging browsers. The browser is not running in incognito but everything is still cleaned up after each job.
The system needs at least three containers: a manager, a worker, and a RabbitMQ instance used to communicate between the two and schedule jobs. Workers can be scaled in order to have multiple concurrent browser sessions.
The management container exposes a web server listening on port 5000. The only required configuration is the secret auth token which will be used by legitimate clients to authenticate to the server (this is a measure to avoid DoS).
The headless can be run with docker compose:
headless:
image: cybersecnatlab/challenge-headless:latest-manager
restart: unless-stopped
environment:
AUTH_TOKEN: supersecret
RABBITMQ_HOST: headless-rabbitmq
RABBITMQ_QUEUE: headless-jobs
depends_on:
- headless-rabbitmq
headless-rabbitmq:
image: rabbitmq:3.11
restart: unless-stopped
headless-worker:
image: cybersecnatlab/challenge-headless:latest-worker
restart: unless-stopped
environment:
RABBITMQ_HOST: headless-rabbitmq
RABBITMQ_QUEUE: headless-jobs
deploy:
replicas: 4
depends_on:
- headless-rabbitmq
The current version of the workers always keeps a browser ready to go. This might use more resources than needed, if you
don't really need a browser ready right now and can wait for it to start up, you can set the environment
variable LEGACY_MODE
to true
in the worker container(s).
The API exposes two endpoints, both are authenticated using the X-Auth
header containing the authorization token for
the headless service:
POST /
allows to create a new job, and it expects a JSON body containing the following values:
key | required | type | description |
---|---|---|---|
actions | :heavy_check_mark: | array | Describes the list of actions to be performed by the headless browser |
timeout | :x: | number | Defines the time in seconds after which the worker will be terminated |
browser | :x: | 'firefox' | 'chrome' | Specifies what browser to use. DEFAULTS TO FIREFOX. See browser support for more information. |
When a request is received, it is put in a queue for workers to execute. The worker opens a new browser instance, executes all the requested actions in order and then fully closes the browser.
GET /jobs/<job_id>
allows to get the job status in JSON format:
key | type | description |
---|---|---|
status | 'queued' | 'running' | 'finished' | Denotes the current status of the job |
queued_at | number | When the job was queued in UNIX seconds |
running_at | number | null | When the job started running on a worker in UNIX seconds |
finished_at | number | null | When the job was finished in UNIX seconds |
Every action is itself a JSON object and contains a type
value, which is used to define what kind of action will be
executed. The available action types are:
Here is an example on how a request could be made to the headless service using curl:
curl -X POST -H 'Content-Type: application/json' -H 'X-Auth: supersecret' --data '{"actions": [{"type": "request", "url": "http://example.com"}]}' http://localhost:5000
What follows is the documentation for each action type:
request
This is used to make the browser request a specific page. Note that the headless browser is set up so that we can also
do requests with methods other than GET, so we can use this action type to submit forms etc. If that's not enough, you
can use the click
and type
actions
Here are the fields for this action:
key | required | type | default value | description |
---|---|---|---|---|
type | :heavy_check_mark: | "request" | Specifies the action type | |
url | :heavy_check_mark: | string | Specifies the URL to request | |
method | :x: | string | "GET" | Specifies the HTTP method used to perform the request |
data | :x: | string | Defines the request body. Both the encoding and the Content-Type are responsibility of the caller. The Content-Type must be provided in the headers field. The Content-Length will be computed by the caller, so there's no need to specify it | |
headers | :x: | dict | {} | Specifies the request headers to be send with the request |
timeout | :x: | number | DEFAULT_REQUEST_TIMEOUT | Specifies the timeout in seconds for the request to execute. If the request fails to load in time or the browser hangs or something else happens preventing the request from finishing, the worker is killed |
Examples:
{
"type": "request",
"url": "http://example.com",
"timeout": 5
}
{
"type": "request",
"url": "http://example.com/login",
"method": "POST",
"data": "username=admin&password=admin",
"headers": {
"Content-Type": "application/x-www-form-urlencoded"
}
}
sleep
This is used to just... wait for a little bit. Can be useful to allow for some JavaScript to execute.
Here are the fields for this action:
key | required | type | default value | description |
---|---|---|---|---|
type | :heavy_check_mark: | "sleep" | Specifies the action type | |
time | :heavy_check_mark: | number | The time in seconds the worker will wait |
Examples:
{
"type": "sleep",
"time": 5
}
{
"type": "sleep",
"time": 3.1415
}
set-cookie
This is used to set a cookie for a specific domain. IMPORTANT: you cannot specify a domain from this action. If you
want to set a cookie for http://example.com you have to first visit http://example.com (with the request
action) and then use this action to set the cookie.
Here are the fields for this action:
key | required | type | default value | description |
---|---|---|---|---|
type | :heavy_check_mark: | "set-cookie" | Specifies the action type | |
name | :heavy_check_mark: | string | Specifies the name of the cookie to set | |
value | :heavy_check_mark: | string | Specifies the value of the cookie | |
httpOnly | :x: | boolean | Left to the browser | Specifies whether the cookie has to be set "Http Only" |
sameSite | :x: | "Lax" | "Strict" | "None" | Left to the browser | Specifies the Same Site configuration for the cookie |
Examples:
{
"type": "set-cookie",
"name": "session",
"value": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwidXJsIjoiaHR0cHM6Ly93d3cueW91dHViZS5jb20vd2F0Y2g_dj1kUXc0dzlXZ1hjUSIsImlhdCI6MTUxNjIzOTAyMn0",
"httpOnly": true
}
delete-cookie
This is used to delete a cookie for a specific domain. IMPORTANT: you cannot specify a domain from this action. If
you want to delete a cookie for http://example.com you have to first visit http://example.com (with
the request
action) and then use this action to delete the cookie.
Here are the fields for this action:
key | required | type | default value | description |
---|---|---|---|---|
type | :heavy_check_mark: | "delete-cookie" | Specifies the action type | |
name | :heavy_check_mark: | string | Specifies the name of the cookie to delete |
Examples:
{
"type": "delete-cookie",
"name": "session"
}
delete-all-cookies
This is used to delete every cookie set in the browser.
Here are the fields for this action:
key | required | type | default value | description |
---|---|---|---|---|
type | :heavy_check_mark: | "delete-all-cookies" | Specifies the action type |
Examples:
{
"type": "delete-all-cookies"
}
click
This is used to click on a DOM element. The element is selected using a CSS selector (such as #login
or .btn[href="/login"]
). The browser automatically waits for it to be present in the document and scrolls the page to
get it into the viewport. When possible, using directly the request
action is the preferred and faster solution for
submitting forms, use this only when DOM interaction is strictly required.
Here are the fields for this action:
key | required | type | default value | description |
---|---|---|---|---|
type | :heavy_check_mark: | "click" | Specifies the action type | |
element | :heavy_check_mark: | string | Indicates the CSS selector used to find the element to click |
Examples:
{
"type": "click",
"element": "button#submit"
}
type
This is used to type an input into a DOM element. The element is selected using a CSS selector (such as #login
or .btn[href="/login"]
). The browser automatically waits for it to be present in the document and scrolls the page to
get it into the viewport. The browser has a 100ms delay between each key press. When possible, using directly
the request
action is the preferred and faster solution for submitting forms, use this only when DOM interaction is
strictly required.
Here are the fields for this action:
key | required | type | default value | description |
---|---|---|---|---|
type | :heavy_check_mark: | "type" | Specifies the action type | |
element | :heavy_check_mark: | string | Indicates the CSS selector used to find the element to click | |
value | :heavy_check_mark: | string | Indicates the input to type into the element | |
delay | :x: | number | 0.1 | Specifies the delay time in seconds between each keystroke. Might be used to simulate a human or to allow for event handlers to execute properly |
Examples:
{
"type": "type",
"element": "#password",
"value": "supersecretpassword"
}
Each request to the server returns a JSON containing a job id. This job id is added to each log generated by the worker. All logs are written to the stdout and stderr of the container.
Logs contains a list every action performed by the browser and a list of all the URLs which have been requested during the job execution (it's nice to see if a request to ngrok/requestbin has been actually sent).
These logs are extremely useful during local challenge development, trust me.
This headless supports both Firefox and Google Chrome. Since the original version of the headless used Firefox, for backwards compatibility Firefox is still the default option.
The feature sets of the two browsers are equivalent, with a small but potentially important difference: Firefox is configured to skip the security warning presented when submitting data from HTTPS to HTTP. The warning cannot be disabled in Chrome. This can be relevant if you serve your web challenges in HTTP and the player is attempting an attack (something like CSRF) from HTTPS (as it's the case with the new default ngrok tunnels). Such an attack would work on Firefox but not on Chrome. While ultimately this is a player issue, you might get a lot of support tickets because " the headless is not working and the challenge is unsolvable". Keep this in mind. Maybe use HTTPS for your challenges if you intend to use Chrome and you fear this might become an issue.
If you want the nitty-gritty details, here are the specific browser versions running on the headless:
browser/driver | version |
---|---|
firefox | 119.0 |
geckodriver | 0.34.0 |
chrome-for-testing | 122.0.6248.0 |
chromedriver-for-testing | 122.0.6248.0 |
Other than the mandatory AUTH_TOKEN
variable, there are additional optional configuration options which can be
specified in the environment, although the default values are already quite sensible:
JOBS_EXPIRATION
(default 30
): jobs that have been queued for more than the specified number of minutes will be
discarded to avoid clogging the queueJOBS_STATUS_TTL
(default 30
): job statuses will be kept in memory for the specified number of minutes since they
were queued and then discardedWORKER_TIMEOUT
(default 30
): if no worker timeout is specifically provided in the HTTP request, this is the
maximum time in seconds the worker is allowed to execute before being killed. Since each action performed by the
worker has its own timeout, the worker timeout might actually be less than this value.WORKER_TIMEOUT_HARD_LIMIT
(default 120
): this is the absolute maximum time in seconds a worker is allowed to run
before getting killed. This puts a cap both to the WORKER_TIMEOUT
value and to the custom timeout passed in the HTTP
requests.DEFAULT_REQUEST_TIMEOUT
(default 10
): This is the default timeout in seconds for a browser request. If the request
takes more than this limit the worker exits.WORKERS_COUNT
(default 4
): This is the number of concurrent workers running inside the containerFIND_ELEMENT_TIMEOUT
(default 5
): This is the maximum time the browser waits for an element to enter the DOM when
using the actions click
and type
docker pull cybersecnatlab/challenge-headless