GitHub and Microsoft have released ReviewBench, an open benchmark dataset and evaluation framework designed to measure the performance of AI code review agents. Available in research preview, the benchmark provides standardized metrics to evaluate how agentic tools inspect pull requests, surface critical issues, and handle trade-offs between noise and coverage.
Representative Pull Request Corpus
To reflect real-world engineering workloads rather than synthetic test sets, ReviewBench was constructed after analyzing 103.9 million public GitHub pull requests. The resulting evaluation dataset features 219 pull requests from 187 open-source repositories across 19 programming languages. While repository size and language distributions mirror overall GitHub statistics, the benchmark deliberately weights sample selection toward medium and larger multi-file pull requests to focus evaluation on substantive code changes.
Dual Metric Families and Verification
ReviewBench establishes its ground-truth findings through a multi-stage process that was independently audited by senior engineers, who achieved a 96.6% agreement rate on labeled findings. The framework measures reviewer performance using six metrics divided into two categories: grounded metrics and augmented metrics. Grounded metrics track standard precision, recall, and F-beta scores against fixed ground truth, with grounded recall serving as the primary headline metric for comparing different systems. Augmented metrics adjust the evaluation denominator when agents identify valid issues not captured in the original dataset.
Configurable Ranking and Production Signals
Because engineering teams maintain different code reviewing priorities, ReviewBench allows users to filter evaluation results by issue severity and category, as well as adjust beta parameters to favor high precision or high recall. GitHub tested the benchmark internally against successive iterations of Copilot Code Review. In one multi-model ensemble experiment, ReviewBench accurately predicted production performance prior to online A/B testing, anticipating an 8.0% increase in addressed comment precision, a 13.6% boost in recall, an 8.0% reduction in per-review cost, and a major shift toward critical findings over low-severity nits.
AI-generated editorial illustration; not a documentary photograph.
Source: ReviewBench: An open benchmark for AI code review
SIGNAL TITANS
Leave a Reply