aurosmruti/promptfoo-runner repository overview

⁠Promptfoo Runner Docker Image

⁠Overview

This project provides a pre-packaged, automated Docker image for running promptfoo's powerful red team security scans. It's designed for developers, security engineers, and MLOps teams who need a consistent and reproducible way to test the safety and security of their LLM APIs.

The container automates the two-step red teaming process:

1. Generate: Creates a set of adversarial attack prompts tailored to your application's purpose.
2. Evaluate: Runs the generated prompts against your specified API endpoint and records the results.

This setup is non-interactive and CI/CD-friendly, making it perfect for integrating into your automated testing pipelines.

⁠Features

• No Local Installation: Run powerful security scans without installing promptfoo or its dependencies. You only need Docker.
• Automated Workflow: Automatically handles the generate and eval steps in a single command.
• Secure & Flexible: API keys and secrets are passed in at runtime. The image itself is clean and stateless.
• CI/CD Ready: Designed for non-interactive use in automated workflows like GitHub Actions, Jenkins, etc.
• Customizable: Fully controlled by your promptconfig.yaml file and environment variables.

⁠Prerequisites

• Docker⁠ must be installed and running on your system.

⁠Quick Start Guide

Follow these steps to get your first scan running in minutes.

⁠1. Pull the Docker Image

Get the latest version of the runner from Docker Hub.

docker pull aurosmruti/promptfoo-runner:latest

(Replace aurosmruti with your actual Docker Hub username if you pushed your own).

⁠2. Create a Workspace

Create a local directory for your test configurations and results.

mkdir my_llm_tests
cd my_llm_tests

⁠3. Create a Configuration File

Inside my_llm_tests, create a file named promptconfig.yaml. This file tells the runner what to test and how to test it.

Copy and paste this template to start:

# promptconfig.yaml
# yaml-language-server: $schema=https://promptfoo-dev.github.io/schemas/v1.14.0/promptfooconfig.json

description: 'Red team scan for my production LLM API'

targets:
  - id: 'https'
    label: 'my-production-api'
    config:
      # ⬇️ 1. EDIT THIS: The URL of the API you want to test.
      url: 'https://your-api-endpoint.com/generate'
      method: 'POST'
      headers:
        'Content-Type': 'application/json'
        # ⬇️ 2. EDIT THIS: The Authorization header for your API.
        'Authorization': 'Bearer sk-your-real-api-key-goes-here'
      body:
        model: 'gemma3'
        prompt: '{{prompt}}'

redteam:
  plugins:
    - id: 'contracts'
      numTests: 10 # Generate 10 test cases for harmful agreements

Important: Remember to add this file to your .gitignore to avoid committing secrets!

⁠4. Run the Scan

Execute the docker run command from within your my_llm_tests directory.

docker run --rm -it `
  -v "${PWD}:/work:rw" `
  -e PROMPTFOO_DISABLE_TELEMETRY=true `
  aurosmruti/promptfoo-runner:latest `
  /work/promptconfig.yaml /work/results.jsonl

What this command does:

• --rm: Deletes the container after it finishes.
• -it: Runs in interactive mode to show you the logs.
• -v "${PWD}:/work:rw": Mounts your current directory (my_llm_tests) into the container's /work directory. This is how the container accesses your config file and writes back the results.
• -e PROMPTFOO_DISABLE_TELEMETRY=true: Skips telemetry and email prompts.
• aurosmruti/promptfoo-runner:latest: The image to run.
• /work/promptconfig.yaml: The first argument, telling the runner which config file to use.
• /work/results.jsonl: The second argument, telling the runner where to save the results.

⁠5. Check Your Results

After the command completes, you will find a new file, results.jsonl, in your my_llm_tests directory with the detailed results of the scan.

⁠Full Example: Using OpenAI for Red Team Generation

Let's say you want to use OpenAI's gpt-4o to generate more sophisticated attack prompts.

1. Update promptconfig.yaml: Add the generators section to specify which model should create the tests.

# promptconfig.yaml
# ... (targets section remains the same) ...

redteam:
  config:
    # Tell promptfoo to use OpenAI for generating the attack prompts
    generators:
      - id: 'openai:gpt-4o'
  plugins:
    - id: 'contracts'
    - id: 'harmful:insults'
      numTests: 5

2. Update the docker run command: Pass your OPENAI_API_KEY as an environment variable using the -e flag.

docker run --rm -it `
  -v "${PWD}:/work:rw" `
  -e PROMPTFOO_DISABLE_TELEMETRY=true `
  -e OPENAI_API_KEY="sk-your-openai-secret-key-goes-here" `
  aurosmruti/promptfoo-runner:latest `
  /work/promptconfig.yaml /work/results.jsonl

The runner will now automatically use your OpenAI key to authenticate and generate the prompts before running the evaluation against your target API.