On-Policy Self-Training: 70B model beats RLHF checkpoint with zero external data

An arXiv paper (2610.32770v1) introduces On-Policy Self-Training, where a model generates topic-conditioned questions, answers and chain-of-thought traces, then learns from its own output without external text. It gains 7.4 points on MMLU (68.2 to 75.6) and 9.1 on GSM-8K, while cutting training compute by 42%.

• MMLU: 68.2 → 75.6 (+7.4 pts) versus the RLHF-tuned baseline
• GSM-8K: 71.5 → 80.6 (+9.1 pts); toxicity rate down 15%
• Training: 1,400 GPU-hours vs 2,400 (−42%); latency 16 ms/token
• Method: reasoning loss aligns hidden states with and without the CoT trace