RLVR Training Makes Chatbots Guess On Harder Problems
Researchers found this reward-based training helped chatbots solve familiar String equations but pushed them toward guessing on harder shifted tests, suggesting the method can teach brittle shortcuts instead of a lasting way to solve problems.
Key Takeaways
- RLVR Generalization causes models to solve in-distribution tasks but revert to brute-force guessing on harder out-of-distribution tests
- RLVR trains a model by scoring answers against automatically checkable outcomes, such as whether an equation is solved correctly
- In the string-equation experiments, models learned a structured derivation on familiar tasks then switched to enumeration on harder shifts, causing accuracy to fall sharply
- Co-training on bijective inversion auxiliary tasks made derivation more certain and restored better strategy use under distribution shift
The researchers wrote that the failure was not simple reward hacking or a weak base model. RLVR, short for reinforcement learning with verifiable rewards, trains a model by scoring answers against outcomes that can be checked automatically, such as whether an equation is solved correctly.
The paper used a controlled string-equation setting where reasoning strategies could be observed directly.
Controlled String Equation Setup
Models learned a structured derivation process on familiar tasks, then dropped that process on harder shifts with more solutions or longer outputs, switching to enumeration and causing accuracy to fall sharply.
That matters because RLVR has become an attractive way to train models on domains where right and wrong answers are easy to grade, math, code and tool use all fit that pattern, and the hope is that verified rewards teach reusable reasoning habits rather than brittle shortcuts.
The authors also linked the paper to open code available. Their main intervention was unusual.
They co-trained the model on bijective inversion tasks that were not uniformly harder and were farther from the target out-of-distribution evaluation, yet that extra training made derivation more certain and restored better strategy use under shift.
Also Read: ChatGPT Spreads To 165 Work Tasks Across 100 Countries
Bijective Cotraining Benefits
The result warns us not to treat clean chain-of-thought traces as proof of strong reasoning. A model can give organized steps on the training distribution, then drop that same strategy when the task structure changes, while the visible format stays similar until the harder setting reveals the shortcut.
This work follows a broader run of research that tries to separate genuine reasoning from pattern matching.
Limits Of Verifiable Rewards
The recent wave of verifiable-reward training has encouraged Optimism (OP) because it avoids some subjective human labeling. But Tuesday’s result narrows that optimism. It suggests that even verified outcomes can teach the wrong habit if the task mix rewards brittle shortcuts. That means RLVR generalization is a systems problem, not just a reward problem, because the training loop, the task family, and the way strategy certainty develops all affect whether a model keeps using a useful method when the world gets less familiar.
How RLVR Generalization Can Fail Quietly
In the paper’s equation experiments, the model could derive answers on easier, in-distribution examples yet fall back to enumeration once the distribution shifted.
The authors interpret this as the model learning success patterns rather than durable procedures.
The bijective co-training auxiliary tasks gave the model settings where derivation was reinforced more reliably. That appears to have strengthened the habit of solving rather than guessing. RLVR generalization improved not because the reward function changed, but because the surrounding task mix changed what the model practiced. This suggests labs should check whether benchmark gains survive first task shifts or hold only in controlled string-equation environments.
Read Next: Simon Willison Says Free Qwen Opens AI Claims To More Checks
