PerceptionBench
PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models
What is it?
What it is
A benchmark designed to evaluate the atomic visual perception capabilities of Multimodal Large Language Models (MLLMs) through 3,000 verified questions across ten perceptual dimensions.
Why it exists
To isolate visual perception from reasoning or domain knowledge errors, addressing the limitations of holistic benchmarks that conflate perceptual failures with higher-level reasoning failures.
Who should use it
Researchers and engineers evaluating multimodal large language models' visual perception capabilities.
Who should avoid it
Beginners lacking machine learning background, users seeking only application-level benchmarks, or non-technical stakeholders.
How it works
A quick walkthrough in plain English
How PerceptionBench works
Step 1 of 3
You interact with it
Open PerceptionBench, send a request, or connect it to your stack.
Features
Advantages
- Isolates atomic visual perception from reasoning or domain knowledge failures
- Provides a high-granularity diagnostic tool for identifying specific model weaknesses
- High alignment with human judgment (99.7% audit agreement)
- Easy to integrate and run via a standardized evaluation script
Disadvantages
- Requires an external LLM judge for grading free-form answers
- Dependency on API keys and network connectivity for evaluation
- Limited to a specific set of 3,000 questions, which may not cover all edge cases of visual perception
Installation
native
git clone git@github.com:MoonshotAI/PerceptionBench.git cd PerceptionBench pip install -r requirements.txt
FAQ
How do I install and run the PerceptionBench evaluation on my local machine?
Clone the repo, install dependencies, configure credentials, and run the eval script: ```bash git clone git@github.com:MoonshotAI/PerceptionBench.git cd PerceptionBench pip install -r requirements.txt cp .env.example .env # then fill in your credentials python eval/eval.py ```
Which environment variables must I set for the evaluation to work?
The following variables are required: - `OPENAI_API_KEY`: Your API key for an OpenAI-compatible endpoint. - `OPENAI_BASE_URL`: Base URL of the endpoint. - `MODEL`: Name of the model you are evaluating. Optional variables include `JUDGE_MODEL`, `DATASET`, `CONCURRENCY`, `MAX_TOKENS`, `MAX_RETRIES`, `PRED_TEMPERATURE`, `TOP_P`, and `TOP_K`. The defaults are listed in the README.
What does the dataset contain and how many questions are there?
PerceptionBench contains 3,000 verified questions covering ten atomic visual perception capabilities. 1,800 (60%) are derived from existing benchmark failures, and 1,200 (40%) are newly authored. The full dataset is available on Hugging Face and can be accessed via the `DATASET` variable.
How should I interpret the leaderboard columns such as VRel, Count, Attr, etc.?
Each column represents accuracy for a specific perception capability: - VRel: visual relation - Count: counting - Attr: attribute detection - Depth: depth & 3D perception - Loc: localization - Comp: comparison - FGR: fine‑grained recognition - Ctx: contextual integration - OCR: optical character recognition - Hallu: perception‑related hallucination. Scores are percentages of correct answers.
Can I evaluate a new model that is not listed on the leaderboard?
Yes. Set the `MODEL` environment variable to your model’s name, ensure it accepts the same prompt format, and run `python eval/eval.py`. Results will be written to `results/PerceptionBench_<model>.jsonl` and a summary JSON will be generated.
Featured in Videos
YouTube tutorials and walkthroughs for PerceptionBench
Alternatives
Similar projects ranked by category, topics, and text overlap.
