NanoGPT sets training speed record at 39.9 seconds, down from 73.9

Deven Pietrzak set a new NanoGPT training speed record of 39.914 seconds, a 46% reduction from the previous 73.889 seconds. The run was reproduced by four independent sources and reached a final cross-entropy of 3.27731.

• Training took 39.914 s, 46% faster than the previous 73.889 s
• Sampled-softmax and hashed n-gram embeddings saved 8 s and 7.05 s
• ANVIL optimizer and FP8 quantization saved 5.08 s and 4.96 s
• Final cross-entropy was 3.27731 with a standard deviation of 0.00099