mattisthegreatest/tempest repository overview

⁠Tempest

Tempest is a YAML-based HTTP API test runner built for readable, composable test suites.

⁠Features

• Declarative HTTP tests in YAML
• CEL⁠ and Liquid Templates⁠ are used for test preparation and assertions
• Automatic retries with flaky test detection
• Built-in console and json reports - or build your own!
• Nested structure with cascading directory and descriptor options

⁠Quick start

Create a directory for your tests and add my-test.spec.yml:

# my-test.spec.yml
name: "Fetch the octocat/Hello-World repo"
test:
  route: "https://api.github.com/repos/octocat/Hello-World"
  headers:
    Accept: "application/vnd.github.v3+json"
  assert:
    - status == 200
    - body.json().full_name == "octocat/Hello-World"
    - body.json().owner.login == "octocat"

Mount that directory at /etc/tests and run the test command:

# Bash
docker run --rm --volume "$PWD:/etc/tests" mattisthegreatest/tempest:latest test

The Docker image supports Linux AMD64 and ARM64.

More executable scenarios are available under examples/tests⁠.

⁠CLI

Tempest currently provides one command:

tempest test [OPTIONS]

⁠Project files and discovery

Tempest recursively scans the directory selected by --path and recognizes these files:

The filename suffix is significant: users.spec.yml is discovered, while users.yml is ignored.

Environment files support simple KEY=value lines. Blank lines are ignored, and # begins a comment even when it appears after a value. These files are project inputs rather than a full shell parser, so shell expansion, export, and complex quoting should not be relied upon.

API_BASE_URI=https://api.example.com
API_TOKEN=secret

test:
  route: "{{ env.API_BASE_URI }}/users"
  headers:
    Authorization: "Bearer {{ env.API_TOKEN }}"

Configuration and environment values discovered in a directory are available to its descendants. Prefer one config and one environment file at each configuration level so precedence remains obvious.

--run filters which specs execute; Tempest still discovers the project from --path so relevant configuration, environment values, and report templates remain available.

⁠Test specifications

A spec file contains a descriptor. A descriptor can contain a test, nested descriptors under describe, or both.

name: "Display name"
description: "Optional longer explanation"
tags:
  - smoke
  - api

options:
  base_uri: https://api.example.com
  retries: 1

describe:
  - name: "Nested section"
    test:
      route: /health
      assert:
        - status == 200

test:
  route: /status
  assert:
    - status == 200

Nested descriptors execute depth-first in source order. Options on a descriptor are inherited by its descendants; options on one sibling do not affect another sibling. tags is currently metadata only and cannot yet be used to select tests.

⁠Loops and profiles

Use options.loop to repeat a descriptor and its subtree a positive number of times. loop is descriptor-local rather than inherited: the parent already repeats its complete subtree, so applying the same loop again to every child would multiply it twice.

name: "Health check {{ iteration.loop_index }}"
options:
  loop: 2
test:
  route: /health

Use a non-empty descriptor-level profiles list to run the descriptor once for each input mapping:

name: "Post {{ profile.post_id }}"
profiles:
  - post_id: 1
  - post_id: 2
test:
  route: "/posts/{{ profile.post_id }}"
  assert:
    - body.json().id == {{ profile.post_id }}

When both are present, profiles are the outer expansion and loop is the inner expansion. Nested profiles form a Cartesian product. A child profile is shallow-merged over the active parent profile; matching child keys temporarily override parent keys, and the parent profile is restored before the next sibling. Expanded cases execute sequentially in source order.

⁠HTTP test fields

test:
  route: /posts?postId=1
  verb: POST
  body: '{"title":"Tempest","userId":1}'
  headers:
    Content-Type: application/json
    Accept: application/json
  let:
    post: body.json()
  assert:
    - status == 201u
    - let.post.title == "Tempest"
  vars:
    post_id: let.post.id

Supported methods are GET, POST, PUT, PATCH, DELETE, and HEAD. An unsupported method currently produces an empty result with status 0 and sends no request.

HTTP status alone does not determine whether a test passes. A test with no assertions is considered passed, even if the response is a 4xx or 5xx response. Write the expected status explicitly.

Transport failures produce an empty response with status 504; the underlying error is available in status_message.

⁠Configuration

Run options can be placed directly in a *.config.yml file:

base_uri: https://api.example.com
debug: false
retries: 1
reports:
  - console
  - json
concurrent: true

The same fields can appear under options on any descriptor:

name: "Eventually consistent endpoint"
options:
  retries: 3
test:
  route: /jobs/123
  assert:
    - status == 200

Options cascade from command defaults through parent-directory configs, child-directory configs, and nested descriptor options. The more local configured value wins.

If a config or descriptor defines reports, include console explicitly if console output should remain enabled:

reports:
  - console
  - json

⁠Liquid input interpolation

Tempest renders spec inputs with Liquid before sending a request. This includes descriptor names and descriptions, base_uri, routes, verbs, bodies, header names and values, assertions, and variable expressions.

Available input globals are:

iteration contains case_index, case_count, profile_index, profile_count, loop_index, and loop_count. All indexes are zero-based.

test:
  route: "/users/{{ vars.user_id }}"
  headers:
    Authorization: "Bearer {{ env.API_TOKEN }}"

CEL and Liquid have different jobs:

• Use CEL to inspect the current HTTP response in let, assert, and on the right-hand side of vars.
• Use Liquid to interpolate environment values, retry state, and values saved by earlier tests.

When a Liquid value is rendered into a CEL string comparison, keep the CEL string quotes:

assert:
  - 'body.json().name == "{{ vars.expected_name }}"'

Numbers and booleans can normally be rendered without quotes:

assert:
  - 'body.json().id == {{ vars.expected_id }}'

Input interpolation and custom report rendering use the same Liquid engine, but each receives a different set of globals.

⁠CEL assertions

Every expression under assert is evaluated against the HTTP response and must return a boolean. false, parse errors, execution errors, and non-boolean results all fail the assertion; evaluation errors are included in reporter output.

⁠Response variables

assert:
  - status == 200
  - status_message == "OK"
  - headers["content-type"].contains("application/json")
  - duration < 1000u
  - body.contains("Tempest")

Normal CEL operators, methods, macros, and literals supported by the bundled interpreter are available. Existing examples use contains, startsWith, endsWith, matches, size, all, exists, exists_one, filter, and map.

⁠JSON

Call .json() on a string to parse it into CEL-compatible JSON data:

assert:
  - body.json().id == 1
  - body.json()["display-name"] != ""
  - body.json().all(item, item.id > 0)

Dotted access works for identifier-like keys; bracket access works for keys containing punctuation. Invalid JSON produces an assertion evaluation error.

⁠HTML and CSS selectors

Call .css(selector) on an HTML string. It returns a list of matches containing tag, normalized text, and an attrs map:

assert:
  - body.css("title").exists(element, element.text == "Example Domain")
  - body.css("a[href]").all(element, element.attrs.href.startsWith("https://"))
  - body.css("main article").size() > 0

An invalid selector produces an assertion evaluation error.

⁠XML and XPath

Call .xpath(expression) on an XML string. Boolean, number, and string XPath results become the corresponding CEL scalar; node sets become a list of strings.

assert:
  - body.xpath("count(/slideshow/slide)") == 2.0
  - body.xpath("string(/slideshow/@title)") == "Sample Slide Show"
  - body.xpath("/slideshow/slide/title") == ["First", "Second"]

⁠Bytes and files

fileBytes(path) reads a file and returns bytes. Relative paths resolve from the current spec file's directory. Paths beginning with / resolve from the suite root selected by --path; paths may not escape that root.

assert:
  - bytes == fileBytes("fixtures/avatar.png")
  - bytes == fileBytes("/shared/avatar.png")

Use .toBase64() to encode bytes and .fromBase64() to decode a Base64 string:

assert:
  - bytes.toBase64() == "aGVsbG8="
  - '"aGVsbG8=".fromBase64() == b"hello"'

Missing files, rejected paths, and invalid Base64 produce evaluation errors.

⁠Test-scoped bindings

Use let to name response-derived CEL values for the current test attempt. Bindings are evaluated in declaration order after the response arrives and before assertions run. Each expression can reference bindings declared above it through the let namespace.

test:
  route: /albums
  let:
    json: body.json()
    album_ids: let.json.map(album, album.id)
  assert:
    - status == 200
    - let.json.all(album, album.title.size() > 0)
    - let.album_ids.size() == 100
  vars:
    first_album_id: let.album_ids[0]

Bindings retain their CEL types, are recomputed for every retry attempt, and do not persist into later tests. Saved-variable expressions in the same test may reference them. If a binding cannot be parsed or evaluated, the attempt fails, later bindings and assertions are not evaluated, and vars are not assigned for that attempt.

⁠Saved variables

Use vars to evaluate response-derived values and make them available to later tests in the same spec file. Each value is a CEL expression evaluated with the same variables, functions, and test-scoped let bindings available to assertions.

name: "Load a user"
test:
  route: /users/1
  assert:
    - status == 200
  vars:
    user_id: body.json().id
    user_name: body.json().name
    response_type: headers["content-type"]

Later descriptors access those values through Liquid's vars object:

name: "Use the saved user"
test:
  route: "/users/{{ vars.user_id }}"
  assert:
    - status == 200
    - 'body.json().name == "{{ vars.user_name }}"'

Variable behavior:

• Producer and consumer tests must be in the same spec file.
• The producer must appear before the consumer in execution order.
• Variables do not cross spec-file boundaries.
• CEL values are converted to JSON-compatible Liquid values.
• If a variable expression fails or cannot be converted, Tempest stores null; assignment failure does not itself fail the test.
• Mutations from a failed retry attempt are rolled back before the next attempt.
• A variable exported under an active loop or profile becomes an ordered array and appends once per terminal test attempt.
• Collected variables bleed into later expanded cases and subsequent descriptors, allowing later criteria to depend on earlier results.
• Once a variable becomes collected, later assignments to that name continue appending so its type remains stable.
• A pre-existing scalar is preserved as the first element when an expanded assignment promotes it to a collected variable.
• Collection indexes follow actual assignment order. Skipped tests and paths that do not evaluate vars do not reserve entries.

profiles:
  - post_id: 1
  - post_id: 2
test:
  route: "/posts/{{ profile.post_id }}"
  vars:
    post_ids: body.json().id

# A later descriptor can use vars.post_ids[0] and vars.post_ids[1].

⁠Retries and flaky tests

retries is the number of additional attempts after an assertion failure. With retries: 1, a test can run at most twice.

options:
  retries: 2
test:
  route: /eventually-consistent-resource
  assert:
    - status == 200

A test that fails and later passes is reported as flaky. Every attempt is rendered, but the final summary counts the descriptor once.

The zero-based retry_attempts Liquid value can vary request inputs by attempt:

options:
  retries: 1
test:
  route: "/status/{% if retry_attempts == 0 %}500{% else %}200{% endif %}"
  assert:
    - status == 200

Flaky tests normally return exit code 0. Use --strict to return exit code 2 when a run contains flaky tests and no failures.

⁠Concurrency

Spec files execute serially by default. Tests within one spec file always remain sequential so saved variables and retries have deterministic ordering.

Enable concurrent spec files in the root project config:

concurrent: true

Without an explicit worker limit, Tempest uses the machine's available parallelism. Use --workers N to set a positive cap; providing --workers enables file concurrency even when concurrent is absent or false.

# At most four spec files at once.
tempest test --path ./tests --workers 4

# Explicitly force serial execution.
tempest test --path ./tests --workers 1

concurrent is a suite scheduler setting, so configure it at the project root rather than on individual descriptors.

⁠Reporters

Tempest includes two built-in report templates:

The console reporter automatically prefixes expanded tests with their one-based location, for example [profile #1/2] [loop #2/3]. Nested expansions include each active profile and loop from outermost to innermost.

Select reporters with the reports option:

reports:
  - console
  - json

⁠Custom report templates

Create a *.template.yml or *.template.yaml file anywhere in the discovered test tree. Its registered name is the lowercase filename without .template.yml; for example, JUnit.template.yml is selected as junit.

# concise.template.yml
title_template: |
  Running {{ test_count }} tests

section_template: |
  # {{ full_name }}

test_template: |
  {% if skipped %}
    {{ expansion_prefix }} {{ full_name }}: skipped
  {% else %}
    {{ expansion_prefix }} {{ full_name }}: {{ status }} {{ status_message }}
  {% endif %}

summary_template: |
  Passed: {{ passed }}, flaky: {{ flaky }}, failed: {{ failed }}, skipped: {{ skipped }}

error_template: |
  Template error: {{ liquid_error_message }}

debug_template: |
  {{ debug_message }}

file:
  dir: ./tempest-reports
  file_name: report-{{ start_timestamp }}.txt

If file is omitted, rendered content is printed to the console. File-backed reporters create their directory when needed and append output to the target file. start_timestamp is available when rendering file_name.

Template fields may contain inline Liquid or refer to a .liquid file beside the YAML template:

test_template: concise.test.liquid
summary_template: concise.summary.liquid

Unknown report names are ignored.

⁠Report template globals

Each item in assertions contains expr, passed, and error.

expansion_prefix is empty for an ordinary descriptor. For expanded descriptors it contains the same one-based profile and loop locations used by the console reporter.

The Liquid engine includes its standard library plus Tempest's json, color_status, and color_duration filters and ANSI color filters such as red, green, yellow, bright_red, and their supported on_* background variants. The built-in templates under tempest/src/builtin_reporters⁠ provide complete examples.

The Liquid json report filter remains valid and is unrelated to the obsolete CEL json response variable.

⁠Exit codes

Warning-as-error handling takes precedence over the flaky exit code.

⁠Current limitations

• Tempest currently runs HTTP tests only.
• tags are metadata and cannot currently filter --run selections.
• File-backed reports append to existing output files.
• The repository examples call public services and therefore require network access.

⁠Developing Tempest

Run the passing examples from the repository:

cd tempest
cargo run -- test --path ../examples/tests --run pass

Run the automated checks:

cd tempest
cargo test
cargo clippy --all-targets --all-features -- -D warnings

⁠License

Tempest is available