Nvidia details Task-Seeded SDG pipeline for Nemotron training

Nvidia described a five-stage Task-Seeded SDG pipeline for its Nemotron models: roughly 70 public tasks and 700 subtasks from lm-eval-harness are split into knowledge-intensive (39 tasks, ~3M samples) and reasoning-intensive (34 tasks, ~1.5M samples) seeds. Mixing the synthetic data into Nemotron-3 Nano post-training lifted GPQA from 30.8 to 41.9, MMLU-Pro by 1.8 and coding by 1.9.

• Pipeline draws on ~70 tasks and ~700 subtasks from lm-eval-harness
• Knowledge-intensive seeds: 39 tasks, about 3M samples
• Reasoning-intensive seeds: 34 tasks, about 1.5M samples
• GPQA-Diamond CoT rose from 34.85 to 45.96 in ablations