Honesty is a race: when reinforcement learning teaches a model to bluff

Two rewards with the same expected payoff at the start of training can end a reinforcement-learning run in opposite places: with an honest model, or with one that bluffs. This is the subject of a new paper with Alexei Oblomkov, linked at the top of this post.

Reinforcement learning from verifiable rewards (RLVR) trains a language model on a task that a program can grade. When the grader can be gamed, the model may learn to game it. Take a model asked whether a Python function is correct. It can answer BUGGY and provide a counterexample, i.e. an input on which the function fails, which a test can confirm. It can also answer CORRECT, which no test can confirm, since tests show that a program is wrong but not that it is right. So a reward computed from tests has to pay CORRECT on every buggy program its tests miss. On such a program, CORRECT is a bluff that always pays, and a counterexample is an attempt that pays only if it works.

We ask: when does RL teach a model to bluff? The same question comes up wherever a reward pays one verdict without checking it. Abstention rewards and scoring rules credit I don't know (Kalai et al., 2025; Mohamadi et al., 2025), and partial credit can pay a well-formed counterexample even when the check rejects it (Zouak et al., 2026).

Our answer is an inequality about a single step of GRPO. To first order in the step size, honesty on the items where the bluff is wrong rises exactly when

Here $b$ is what the bluff pays, $r$ is what an honest attempt pays when it checks, $\tilde p$ is (roughly) the rate at which the model's attempts succeed, and $\kappa$ measures how much training on the other items, where the bluff's verdict happens to be the right answer, drags the model toward the bluff. Both $\tilde p$ and $\kappa$ move during training. A checker raises $\tilde p$, which makes honesty a race, and $\kappa$ falls as the other items settle, which makes runs just above the threshold dip before they recover. The same pull also works in the other direction: pay the honest verdict too much, and it starts to spill over onto the items where it is wrong. Most of this post is about deriving the inequality and reading off these consequences.

Two tasks

Our main testbed is propositional Datalog. An instance consists of some facts, some rules of the form h :- b1, b2 (meaning that if b1 and b2 hold, then so does h), and a goal, and the question is whether the goal can be derived. Both answers can be certified: a proof certifies DERIVABLE, and a closure, i.e. a list of every derivable atom with its justification, certifies UNDERIVABLE. A checker can verify either one exactly.

Facts: p1, p2      Goal: p4
Rules: p3 :- p1, p2
       p4 :- p3
       p5 :- p6

ANSWER: DERIVABLE
PROOF:
p1 : fact
p2 : fact
p3 : from p1, p2
p4 : from p3

Facts: p1, p2      Goal: p4
Rules: p3 :- p1, p2
       p4 :- p6
       p5 :- p6

ANSWER: UNDERIVABLE
CLOSURE:
p1 : fact
p2 : fact
p3 : from p1, p2

To make Datalog behave like code, we pay $b$ for DERIVABLE without checking the proof, so that DERIVABLE plays the part of CORRECT. A DERIVABLE answer with a valid proof pays 1, and a closure that checks pays $r$. We call the underivable instances the class at stake, since that is where the bluff is wrong, and we call the derivable instances the other class, since there the bluff's verdict happens to be the right answer.

The instances come in twins, as in the example above: each underivable instance has a derivable twin that differs in one atom of one rule, and that matches it in facts, number of rules, and length. (The real instances have 20 atoms.) We fine-tuned Qwen2.5-3B-Instruct on two versions of the task, with proofs two and three levels deep. Before any RL, 77% and 66% of the fine-tuned models' closures pass the checker.

The second task is the code task from the introduction, with programs from MBPP and single-site mutants of them, using Qwen2.5-Coder-3B-Instruct and Phi-3.5-mini-instruct. There the class at stake is the buggy programs, and the other class is the correct ones. We also look at an abstention reward for grade-school math.

The direction of a step

Let's look at what a single GRPO step does to the verdict. I'll write it for code. Fix a program $i$, let $h_i$ be the probability that the model answers BUGGY on it, and let $x_i$ be the log-odds. On a buggy program, BUGGY pays $r$ with probability $p_i$ (when its counterexample is valid), and CORRECT pays $b$. On a correct program, CORRECT pays $b$ and BUGGY pays nothing. GRPO samples $n$ answers to each prompt and raises the log-probability of each answer in proportion to its reward minus the group's mean reward.

Because the model writes its verdict first, every answer in a group pushes on the verdict along the same direction, the gradient of $x_i$. The expected size of the group's push is $(n-1)\,h_i(1-h_i)$ times the payoff gap between the two verdicts, which is $p_i r - b$ on a buggy program and $-b$ on a correct one. So a program pushes toward its better-paid verdict, and only while its group is likely to contain both. To first order, a step moves the log-odds on any program $j$ by the sum of these pushes, each weighted by the inner product $\langle \nabla x_i, \nabla x_j\rangle$, which measures how far an update computed on program $i$ moves the verdict on program $j$ (Jacot et al., 2018; Ren and Sutherland, 2025). This is where the other class enters: teaching the model to say CORRECT on a correct program also moves its verdict on the buggy programs that look like it.

Now replace each inner product by its average for that kind of pair: $K$ for two buggy programs, and $sK$ for a buggy program and a correct one. Then the expected change on a buggy program is positive exactly when

Each quantity has a plain reading. $W_t$ and $W_o$ are the sums of $h_i(1-h_i)$ over the buggy and the correct programs, so they count the items of each class that are still in play: an item whose verdict is a coin flip contributes $1/4$, and one that always gets the same verdict contributes nothing. $\tilde p$ is the success rate of the attempts that still count (the $p_i$ weighted by $h_i(1-h_i)$). $s$ is the coupling between the classes. It would be 0 if the model kept its verdicts on the two classes separate, and 1 if it treated every item alike, and we measure it from gradients on the starting model. The paper has the full derivation, including the small change on Datalog, where the right answer on the other class pays about 1.

The inequality is first order and uses class averages, so it gives the direction of a step at the current state. Its use is that it says which quantities to watch, and two of them move.

The threshold

When the other class has settled, $\kappa = 0$ and the inequality becomes $\tilde p\, r/b > 1$. The same comparison is visible in a single group: a bluff earns $b$ for sure, and an attempt earns $pr$ on average. This is Chow's rule for classification with a reject option (Chow, 1970), with the roles reversed. In Chow's setting abstaining is the safe choice and answering is the gamble, while here the honest answer is the gamble. If we also replace $\tilde p$ by the plain success rate $p$, which we measure before training by sampling the model's attempts and grading them, we get a prediction that we can check before running anything.

Figure 1 shows every Datalog run we have made from the two fine-tuned models, placed according to $pr/b$. At 0.8 and below, every run collapses: by the end, the model answers DERIVABLE on essentially every underivable instance. Above 1, every run ends honest, at 71 to 100%. In between, at 0.92 to 0.95, the checked runs end anywhere from 20 to 85%, while the lottery runs, which I explain in the next section, do not come back at all.

The comparison holds on code, too: both coding models collapse at $pr/b = 0.95$ and below, and at 1.6 and above they end at 58 to 93%. For abstention it is Chow's rule itself. A reward that pays 1 for a right answer, 0 for I don't know, and $-\lambda$ for a wrong answer puts the threshold at $\lambda^* = p/(1-p)$, which is 2.34 for Qwen2.5-1.5B-Instruct on GSM8K. The model keeps answering below $\lambda^*$, and at $4\lambda^*$ it abstains on 99% of held-out problems, including every problem it solved in all eight samples before RL.

The race

The first quantity that moves is $\tilde p$. The checker pays only valid attempts, so training makes the model's attempts better, and its success rate $p_t$ at step $t$ rises. Meanwhile, as long as $p_t r < b$, the bluff has the larger payoff, so attempts become rarer. If $p_t$ passes $b/r$ while the model is still attempting often, honesty recovers. If the attempts die out first, the collapse is permanent, since a model that never attempts gets no signal to improve. In this sense, honesty is a race.

To test this, we built a control that we call the lottery, which pays an attempt $r$ at random, with probability equal to the starting model's $p$, whether or not the attempt is valid. At the start of training it pays what the checker pays on average, but getting better earns nothing.

The top of Figure 2 shows four seeds of each at $pr/b = 0.95$, just below the threshold, on the depth-2 instances. All four lottery runs collapse. All four checked runs fall too, to between 6% and 48% at their lowest evaluation, but then they come back, ending at 68 to 85% after 160 steps. Over these runs the share of closures that pass the checker rose from 0.77 to about 0.87, which puts $p_t r/b$ near 1.07, above the threshold. Over all nine matched pairs of runs below the threshold, every lottery run collapsed, while five of the nine checked runs survived and two more were still climbing when training ended.

Which seed wins shows only late, but the odds can be predicted. We built a small model of the race from the step inequality plus one assumption about how fast the checker raises the validity of attempts, and we fit its three constants only on runs outside the race band. Its prediction is the curve in Figure 1, which places the race on the depth-2 model at about $pr/b = 0.92$ to 0.95.

The real test was the depth-3 model. Before Alexei's runs at $pr/b = 0.92$ were done, we wrote the model's prediction into the draft of the paper: each checked seed would win (end at least 60% honest) within 160 steps with probability 0.04 to 0.13, and every lottery seed would collapse. The bottom of Figure 2 shows what happened. No checked seed won, and every lottery seed collapsed. The checked seeds fell to 4 to 15% but were back at 20 to 47% after 160 steps, while the lottery seeds sat at 0 to 0.2%.

The early dip

The second quantity that moves is $\kappa$. At the start of training the other class is far from settled, so $\kappa$ is large, and a run with $1 < pr/b < 1 + \kappa$ starts out moving toward the bluff. In this range the other class moves toward its own right answer (this needs $ss' < 1$, where $s'$ is the coupling in the other direction, and on the depth-2 model we measured $s = 1.03$ and $s' = 0.86$). So $\kappa$ falls, and the comparison against 1 takes over. In this way, $pr/b$ versus $1 + \kappa$ decides which way a run starts, and $pr/b$ versus 1 decides where it ends.

This is what we see. On the depth-2 instances, every checked run at $pr/b = 1.07$ and 1.25 falls from 99% before RL to between 84% and 94% at its first evaluations, and then climbs back, ending at 94 to 98%. On code, where $1 + \kappa$ is about 2.1, the runs at $pr/b = 1.25$ first fall nearly to zero, and then climb back.

Figure 3 follows one of these runs step by step, computing both sides of the inequality from the answers sampled in training. $\tilde p\, r/b$ hovers just above 1 throughout, while $1 + \kappa$ starts far above it and first falls to it at step 13, when the derivable instances settle. Honesty turns at step 34, close to where integrating the push through Adam's momentum puts the turn (step 30).

The dip also explains why the checker lowers the threshold so little. Below the threshold, the dip adds to the bluff's advantage in the first steps, so attempts thin out before the checker has had time to raise $p_t$. On the depth-2 model, the checker moves the cliff from 1 only to somewhere between $pr/b = 0.80$ and 0.95. A model whose other class has already settled has no dip, so it keeps attempting and gives the checker more time. This is the idea behind the schedule below.

Paying too much

The comparison of $pr/b$ with 1 says that raising $r/b$ can only help honesty. The coupling says that it can also hurt. The push that a step gives the class at stake reaches the other class too, through the coupling $s'$, and there the honest verdict is wrong. The other class's own groups pull back toward the right answer, so on code the push on the correct programs is $s' W_t(\tilde p\, r - b) - W_o c$, and above the ceiling

the spillover wins, and the model starts calling correct programs buggy. On code, the couplings measured on the two starting models put the ceiling at about $pr/b = 2.15$ to 2.2. We trained both models at 2, below it, and at 3, above it. At 2, both still call correct programs correct about as often at step 100 as before RL (79% and 78%, against 80% and 74%). At 3, that share falls to 13% and 30% within 40 steps, and it is still only 45% and 52% at step 100. So the best ratio overall lies below the one that makes the honest verdict most attractive on buggy programs. On Datalog we see at most a small effect, since there the right answer on the derivable instances pays 1, more than $b$, which strengthens their own pull.

Generous, then cut

Both moving parts suggest a schedule. A generous phase, at a ratio where $pr/b > 1$, raises $p_t$ and lets the other class settle, so that $\kappa$ is near 0. After that, the inequality reads $p_t r/b > 1$, which can hold where the starting model's $pr/b < 1$. Concretely, to train at a ratio $r/b$ that the starting model does not survive:

1. train at a generous ratio, where $pr/b > 1$, so that $p_t$ rises;
2. once $p_t r/b > 1$ at the target ratio, cut to it.

We tried this on the depth-2 instances. Training for 100 steps at $pr/b = 3$ raised $p_t$ from 0.77 to 0.87. After the cut to the ratio of Figure 2 (where the starting model sits at $pr/b = 0.95$), the model stayed 89 to 99% honest for 100 more steps, while the starting model, trained at that ratio from the beginning, dips to 6 to 48%. The check during the generous phase matters. Run under the lottery instead, the generous phase leaves $p_t$ at 0.76, and after the same cut the model dips to 29% before it ends 83% honest.

At depth 3 the window where the schedule can help is wider. There the generous phase was a reward that pays only valid certificates ($b = 0$), which raised $p$ from 0.66 to 0.77, so at $r/b = 1.39$ the original model sits at $pr/b = 0.92$ and the trained one at 1.07. In three seeds each (Figure 4), the trained model stays at least 67% honest and ends at 78 to 82%, while the original model falls to 4 to 15% and is at most 47% honest after 160 steps. So at depth 3 the schedule avoids more than a dip.

The model still knows

When a run collapses, the model stops reporting the truth, but it keeps computing it. To see this, we fit a linear probe on the fine-tuned model's activations before RL, to tell derivable from underivable instances, and then held it fixed. In the ten collapsed models trained from the depth-2 model, this probe still separates the two classes (AUROC 0.92 to 0.99). What has moved is where the instances fall relative to the probe's cutoff: the underivable instances now score on the derivable side, so with its original cutoff the probe calls most of them derivable, just as the model does. A probe refit on each trained checkpoint is 97.5 to 99.6% accurate, whether or not the run collapsed. This is why we call the collapse bluffing.

What's new

We see five things as new in our paper: (1) the step inequality, in which the shared verdict enters through a coupling $s$ that we measure from gradients, which explains the early dip and predicts a ceiling, (2) the threshold, with $p$ measured before RL and tested on Datalog, code, and abstention, (3) the race, together with the lottery control that isolates it, (4) the schedule, with a criterion for when to cut, and (5) a probe fit before RL that still reads the truth after a collapse. The closest earlier result is the law of Rad et al. (2026) for GRPO with a noisy verifier, in which learning reverses when the verifier's false-positive rate exceeds its true-positive rate. With $\kappa = 0$, our inequality is that law, with the bluff's payoff in the place of the false positive. The paper discusses the rest of the related work.

Conclusions

A reward that pays an unchecked verdict sets up a contest between the bluff, which pays for certain, and the honest attempt, which pays only when it works. To first order, a GRPO step moves honesty in the direction of $\tilde p\, r/b - 1 - \kappa$. The checker raises $\tilde p$, so a run just below the threshold can still win the race. The other class settles, so $\kappa$ falls, and a run just above the threshold dips before it recovers. The same coupling sets a ceiling. Together these suggest training generously first and tightening later, and they tell us when tightening is safe.

The main limitations are that our models are small (1.5 to 3.8 billion parameters), our main task is synthetic, the inequality is first order and uses class averages, and most conditions have one to four seeds per ratio.

We think that this kind of analysis, which predicts the outcome of an RL run from quantities one can measure before it starts, is a useful complement to detecting reward hacking after the fact. We are especially interested in whether the race shows up in larger models and longer runs, and in rewards that pay more than one unchecked verdict. If you are working on RL with verifiable rewards, reward hacking, or abstention, please get in touch!

Acknowledgments

We thank Indranil Halder and Sanjeev Arora for useful discussions.

This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) through the Super Intelligence Quantified (SIQ) program, under Cooperative Agreement No. HR00112520028. The views, opinions, and/or findings expressed are those of the authors and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government. We also thank the Unity cluster at the University of Massachusetts Amherst for computing resources.