am1n3e/webarena-verified repository overview

⁠WebArena-Verified

WebArena-Verified is the verified release of the WebArena benchmark. It distributes a curated, version-controlled dataset of web tasks together with deterministic evaluators that operate on agent responses and captured network traces.

⁠Quick Start

docker pull am1n3e/webarena-verified:latest

Run a single evaluation:

docker run --rm am1n3e/webarena-verified evaluate --task-id 0 --answer "your answer"

⁠Documentation

For full documentation, installation options, and usage guides, visit the main repository:

https://github.com/ServiceNow/webarena-verified⁠

⁠Features

• Fully audited benchmark with manually reviewed tasks and evaluators
• Offline evaluation using network trace replay
• Deterministic scoring without LLM-as-a-judge
• WebArena-Verified Hard subset (258 tasks) for cost-effective evaluation

⁠Links

• Documentation: https://servicenow.github.io/webarena-verified/⁠
• PyPI: https://pypi.org/project/webarena-verified/⁠
• Source Code: https://github.com/ServiceNow/webarena-verified⁠

⁠License

See the LICENSE⁠ file in the repository.