GitHub launched ReviewBench on October 5 to make AI code reviewers easier to compare. Its research preview covers 219 public pull requests across 19 languages, with findings organized by severity and category. For a team considering automated review, the useful shift is from asking how many comments an agent produces to inspecting which problems those comments actually identify. GitHub's announcement describes the benchmark, not a guarantee about your repository.
Measure signal and omissions separately
GitHub assembles candidate findings from human reviews, models, and static analysis, then deduplicates and judges them against a shared rubric. Grounded metrics compare a review with the existing reference findings; augmented metrics also assess previously unlisted findings. The company recommends grounded recall for cross-system comparison because augmented recall changes its denominator with each agent's discoveries. The methodology explains that distinction.
AyNcode's analysis: separate two questions before buying or building a reviewer. How much of its output deserves an engineer's attention? And which important defects does it miss? A single headline score can conceal a trade-off your team would not accept.
A worked example: fewer comments can be better
Hypothetical pilot, not a ReviewBench result: suppose a manually checked set of changes contains 12 distinct defects. Reviewer A flags ten issues, eight of them valid. Reviewer B flags seven, all valid. Count each underlying defect once, even when the agent mentions it repeatedly.
- Reviewer A: precision is 8/10, or 80%; recall is 8/12, about 67%. It produces two false alarms and misses four known defects.
- Reviewer B: precision is 7/7, or 100%; recall is 7/12, about 58%. It produces no false alarms but misses five known defects.
A catches one extra defect; B creates less investigation work. Neither wins automatically. If A's additional finding prevents an authorization failure, that may outweigh its noise. If it catches only a minor inconsistency, the decision may reverse. Inspect the missed and additional findings, not just the percentages.
Turn the comparison into a rollout decision
Start with a small sample of your own changes: booking imports, permission checks, retries, or subscription handling. Include changes with no known defect, so a reviewer cannot look successful merely by criticizing everything. Ask an engineer to establish expected findings before seeing the agent's comments, then independently investigate unmatched findings rather than assuming the reference list is complete.
Keep the commit, supplied context, reviewer configuration, and available tools fixed when comparing runs. Record time spent verifying comments alongside model and tool charges. Inspect security-related misses separately from style suggestions. Use those records to decide whether to widen coverage, reduce noisy categories, or keep the reviewer advisory.
For Copilot specifically, GitHub's documentation says approvals are off by default; when enabled, its approving review can satisfy a required-approval rule. Our recommendation is to retain a human merge gate for sensitive changes during the pilot. A quiet review is not evidence that the change is safe.
Source context
The launch details and benchmark design above are GitHub's account. Public-repository results and model-assisted judging are reasons to validate locally, not to assume the ranking transfers unchanged. The example and rollout method are AyNcode's analysis.
Image: original AI-generated editorial illustration for AyNcode, not a GitHub interface or evidence of benchmark results.