Meta’s AI Code Checker Was Barely Better Than Chance, Then Its Engineers Tuned It

Meta engineers say an AI judge grading code from their own coding agent agreed with real outcomes only slightly better than chance, until a new method called AutoRefine rewrote its instructions and more than tripled its score from 0.16 to 0.51.

Key Takeaways

• Meta’s AI code judge scored 0.16 on agreement with whether code changes were merged or abandoned, where 0 means no better than chance
• AutoRefine rewrote the judge’s prompt using signals teams already have, like passing tests and merged changes, lifting the score to 0.51
• On a separate benchmark with a reliable answer key, the same method raised agreement from 0.36 to 0.60
• The big jump came from just 17 test changes, and on 16 older ones the previous judge still did better

Eight researchers at Meta Platforms, including co-lead authors Shervin Shahidi and James Poon, described AutoRefine in a paper posted on Nov. 6 and prepared for the industry track of the CAIN 2027 software engineering conference.

Why AI Code Judges Go Stale

Coding agents now write changes that no automated test can fully check, so teams use an AI model as a judge to decide whether a change is good. The paper says that judge’s instructions need tuning for each task and retuning every time the agent changes, while hand-labeled examples are costly and quickly go out of date.

How AutoRefine Works

Development teams already get free signals. Tests pass or fail, and code changes get merged or abandoned. Those signals are noisy, so AutoRefine adds a teacher model that sees both the change and how it turned out. The teacher explains each outcome it keeps and skips any it cannot explain from what the judge sees. The system then rewrites the judge’s prompt to match the outcomes it kept.

In a test where the researchers flipped 25% of the labels at random, the teacher threw out the fake failure labels at more than twice the rate of real ones, cutting overall noise from 25% to 20%.

Also Read: GitHub Guide Puts Copilot Across 4 Coding Jobs, Humans Decide

The Catch

The jump from 0.16 to 0.51 came from 17 held-out code changes, a small sample with a wide margin of error. On 16 changes from older workflows, the previous judge stayed ahead, which the authors say shows a judge must be retuned as its agent evolves.

The paper says the skipped cases are useful on their own, because they tell a team when a judge needs better information rather than a better prompt.

Read Next: Anthropic’s Claude Code Has Been Sending Users’ Email Addresses To Websites