LE CRITIQUE: Privileged Value Functions
for LLM Reinforcement Learning

Siddarth Venkatraman1,2,3 Matthieu Dinot1 Laurence Aitchison1
1Mistral AI 2Mila - Quebec AI Institute 3Université de Montréal

Correspondence: siddarth.venkatraman@mila.quebec

Paper arXiv Code
A privileged value function receives the policy-visible question and partial chain of thought, plus critic-only clues such as a known solution or leave-one-out group responses, then computes a token-level advantage.

A privileged critic sees the policy state plus admissible training-time clues, producing a more informative token-level baseline without exposing those clues to the policy.

OVERVIEW

Stronger critics for LLM reinforcement learning

Group-relative methods such as GRPO avoid a learned critic, but they provide only sequence-level credit and require multiple rollouts per prompt. Learned value functions offer token-level advantages, but a poorly fit critic can make training worse.

Le Critique develops two complementary strategies: Privileged Value Functions (PVF) give the critic useful context hidden from the policy, while TETHER adaptively interpolates between the dependable group baseline and the learned token-level value baseline.

01

Privileged Value Functions

Route admissible training-time information through the critic to improve value estimation and reduce policy-gradient variance.

02

TETHER

Fit an adaptive mixture of leave-one-out group mean and learned value baselines so the critic contributes only as much as its accuracy warrants.

METHOD 1

Privileged Value Functions

A PVF conditions on the ordinary policy history $h_{i,t}$ and additional critic-only context $z_{i,t}$. If this context is admissible, the stronger baseline can reduce variance while leaving the expected policy gradient unchanged.

\[ \begin{aligned} V^\pi(h_{i,t},z_{i,t}) &= \mathbb{E}[R_i \mid h_{i,t},z_{i,t}], \\ \widehat A^{\mathrm{PVF}}_{i,t} &= R_i-V_\phi(h_{i,t},z_{i,t}). \end{aligned} \]

What can be privileged?

01

Reference solutions

Oracle answers, proof sketches, gold patches, or other fixed target information that helps judge a partial trajectory.

02

Leave-one-out groups

Other independently sampled responses and their rewards, turning value prediction into a relative in-context comparison.

03

Task-side context

Verifier rubrics, detailed task specifications, or latent environment information that satisfies the causal guardrail.

Not admissible: future tokens, the current response's realized reward, or later verifier and environment feedback produced by that response.

OBJECTIVE PRESERVATION

PVF and self-distillation use privileged information differently

PVF

Privileged critic

The extra information is used to condition a value function. Under the admissibility condition and with Monte Carlo advantages, it can reduce variance without changing the expected policy gradient or the original RL optimum.

Tradeoff: it cannot use feedback generated from the completed response itself.

SELF-DISTILLATION

Privileged teacher

The extra information shapes a teacher distribution and introduces a distribution-matching objective, typically a token-level reverse KL. This can change the policy optimum and requires careful information and stability control.

Advantage: it can directly exploit retrospective critiques or verifier feedback.

METHOD 2

TETHER: an adaptive group-value baseline

TETHER starts from the reliable leave-one-out group baseline and moves toward token-level value estimates only when the critic improves return prediction. The mixture coefficient $\rho$ is fit by least squares to minimize return prediction and smoothed across batches.

TETHER adaptively mixes the leave-one-out group baseline and current token value using a coefficient rho, then computes the policy advantage from the mixed baseline.
\[ \begin{aligned} b^{\mathrm{TETHER}}_{i,t} &= (1-\rho)b_i^{\mathrm{LOO}}+\rho V_{i,t}, \\ A^{\mathrm{TETHER}}_{i,t} &= R_i-b^{\mathrm{TETHER}}_{i,t}. \end{aligned} \]
$\rho=0$
group baseline $\rho=1$
value baseline
No own-return leakage. Each batch uses the mixture coefficient fixed before that batch; its completed returns update $\rho$ only for the next batch.

EXPERIMENTS

Reasoning tasks and privileged context

PVF is evaluated across three environments. TETHER uses the same suite, omits no-group Reasoning Gym, and adds MiniF2F.

PVF + TETHER

Reasoning Gym

A weighted mixture of procedural, single-turn reasoning tasks.

PVF clue: ground-truth answer
PVF + TETHER

CodeIO

Single-turn program reasoning from inputs to outputs and outputs to feasible inputs.

PVF clue: leave-one-out responses and rewards
PVF + TETHER

Sudoku

A multi-turn environment where the policy reasons and fills one missing grid cell per turn.

PVF clue: complete solved grid
TETHER ADDITION

MiniF2F

A multi-turn Lean formal-mathematics task with compiler feedback across attempts.

Baseline: adaptive group-value mixture

RESULTS

Training rewards

PVF

Best performing method in all four PVF settings.

TETHER

Improves over the ordinary value baseline in all four TETHER settings.

Four seed-averaged reward curves comparing mean, ordinary value function, and privileged value function baselines on Reasoning Gym at K equals 1 and K equals 8, CodeIO, and Sudoku.
Privileged Value Functions. Seed-averaged, EMA-smoothed training reward; shading is one standard deviation. Reasoning Gym uses two seeds; CodeIO and Sudoku use three.
Four seed-averaged reward curves comparing mean, ordinary value function, and TETHER baselines on Reasoning Gym, CodeIO, Sudoku, and MiniF2F.
TETHER. Seed-averaged, EMA-smoothed training reward; shading is one standard deviation. Reasoning Gym uses two seeds; CodeIO, Sudoku, and MiniF2F use three.

Final-window comparison

Bar charts of mean reward over the last 50 policy steps for PVF and TETHER experiments, comparing group mean, ordinary value function, PVF, and TETHER baselines.
Mean raw reward over the final 50 policy steps, averaged within seed and then across seeds. Error bars show one standard deviation across per-seed means.

LE CRITIQUE

Value functions help, when used correctly

PVF strengthens token-level baselines with admissible context. TETHER keeps value functions connected to a reliable group baseline while the critic learns.

Read the paper Explore the code