What is it?
What it is
A stateful agent benchmark for real-world commerce workflows, evaluating agents' ability to complete long-horizon tasks across domains like browser operations, API/MCP workflows, document production, and logistics using local mock services.
Why it exists
To assess practical application of agents in commerce scenarios, ensuring they can handle stateful, multi-step tasks without requiring production accounts or real credentials.
Who should use it
Researchers and developers building or evaluating stateful AI agents for long‑horizon commerce and business workflows, especially those interested in reproducible, auditable benchmarks with mock services and deterministic verifiers.
Who should avoid it
Individuals or teams looking for simple question‑answering benchmarks, or those without access to Docker, Python 3.11+, or the required LLM API keys; also those unwilling to handle the complexity of setting up mock services and harness configurations.
How it works
A quick walkthrough in plain English
How RealReplicaBench works
Step 1 of 3
You interact with it
Open RealReplicaBench, send a request, or connect it to your stack.
Features
Advantages
- Realistic simulation of long-horizon business workflows with state changes
- Transparent and auditable evaluation process with deterministic grading
- Flexible model evaluation across various providers and architectures
- Comprehensive task coverage enabling thorough agent capability assessment
- Active community and collaboration opportunities with the Accio team
- Open-source nature with clear licensing for both code and task definitions
Disadvantages
- Complex setup requirements (Docker, Python 3.11+, API keys)
- High resource consumption due to stateful evaluations and containerized tasks
- Dependency on specific API keys and provider integrations
- CC BY 4.0 licensing for task definitions requiring attribution
- Limited public access to raw task results (stored via leaderboard only)
- Potential learning curve for configuration management and harness customization
Installation
docker
docker pull --platform linux/amd64 acciolyk/accio_bench@sha256:1e9cf5c72a56794175b7d06ece036b92e296e6b7e9e9a7fa244026f6acea3859
FAQ
What is RealReplicaBench and what does it evaluate?
RealReplicaBench is a stateful agent benchmark that tests whether an AI agent can complete long‑horizon business workflows—such as product publishing, freight booking, and storefront customization—by interacting with realistic mock services that mimic real commerce and business software. Each of the 107 tasks runs in a fresh container and is graded by a deterministic or LLM‑assisted verifier.
How do I run a single task from the command line?
After installing the package and pulling the pinned OpenClaw runtime image, you can run a task with a command like: ```bash real-replica-bench run api-amazon-margin-floor-audit \ --harness openclaw \ --image acciolyk/accio_bench@sha256:<digest> \ --platform linux/amd64 \ --openclaw-model google/gemini-3.5-flash \ --openclaw-image-model google/gemini-3.5-flash \ --openclaw-models-config configs/realreplicabench_native_google_direct_models.json \ --llm-judge-provider gemini \ --llm-judge-model gemini-3.1-pro-preview \ --run-id realreplicabench-smoke ``` Replace the task ID and model choices as needed.
What Docker image do I need and how do I pull it?
The benchmark uses a pre‑built OpenClaw runtime image that contains the browser stack, mock services, and the OpenClaw runner. Pull it with: ```bash docker pull --platform linux/amd64 \ acciolyk/accio_bench@sha256:1e9cf5c72a56794175b7d06ece036b92e296e6b7e9e9a7fa244026f6acea3859 ``` The digest is fixed for release v1.3.1, ensuring reproducible runs.
Which credentials do I need to provide for the agent and the judge?
The agent and judge each require an API key for the chosen provider: * **Agent** – depends on the route: for native Gemini use `GEMINI_API_KEY`; for native Qwen/DashScope use `DASHSCOPE_API_KEY`; for OpenRouter use `OPENROUTER_API_KEY`; for custom endpoints use the appropriate env var such as `OPENAI_API_KEY` or `CUSTOM_GEMINI_API_KEY`. * **Judge** – typically uses Gemini `generateContent`, so set `GEMINI_API_KEY` (or the judge’s own key if you use a different provider). All credentials are passed via environment variables; the runner will fail if any placeholder remains unresolved.
How can I contribute a new mock environment to the benchmark?
To add a new mock, implement a mock service that mimics a real API’s semantics, state transitions, and error handling. Register it in `real_replica_bench/mock_services/registry.py`, create corresponding tasks under `datasets_domain_v1/`, and follow the guidelines in `CONTRIBUTING.md`. Once merged and the runtime image is rebuilt, the new tasks become part of the published benchmark.
How can I get my own model evaluated on RealReplicaBench?
The Accio team runs models on request, including pre‑release builds. Contact them via email at `[email protected]` or `[email protected]` to request a private evaluation or collaboration on new task domains, mock environments, or harness work.
Featured in Videos
YouTube tutorials and walkthroughs for RealReplicaBench
Alternatives
Similar projects ranked by category, topics, and text overlap.
