Hugging Face Journal Club: Score Centering Stabilizes Off-policy Reinforcement Learning
Score centering with top-K approximation stabilizes off-policy LLM reinforcement learning by correcting sampler-trainer policy mismatch.
A Hugging Face Journal Club session breaks down a paper identifying policy mismatch (P-theta ≠ Q) as the root cause of instability in LLM reinforcement learning. The authors derive a score-centering correction term and implement it via a top-K approximation to make it computationally feasible. This is a technically substantive finding for ML practitioners working on RLHF pipelines, but it is a narrow research walkthrough rather than a broad industry signal.