The silent failure in long RL runs: trainer–inference mismatch
Research8 mins read
Introduction
In on-policy reinforcement learning the trainer assumes the samples came from the policy it's optimising. In practice, rollouts are generated by a separately optimised inference engine, and the two rarely agree to the last decimal.
Why it matters
Small disagreements in log-probabilities compound across long sequences. The run doesn't crash; rewards still go up for a while. But the policy drifts toward behaviour the trainer never actually measured.
Detection
Cheap checks on every step
We recompute a sample of rollout log-probs with the training engine and track the KL divergence between the two. A rising mismatch triggers an alert before the damage is baked into a checkpoint.
We pair it with numerical stability checks and load tests on the serving path, so researchers can focus on algorithms rather than plumbing.
