Q Labs' Dust pretrains transformers without backpropagation

Q Labs' open Noise speedrun benchmark shows its Dust algorithm pretraining transformers with forward passes only, reaching validation loss 3.338 versus the 3.340 ANVIL II backpropagation baseline on Track One and 3.211 on the open-architecture track. Dust estimates gradient direction by injecting noise into per-token activations instead of using the chain rule.

• Dust hit 3.338 loss on Track One, edging past the 3.340 ANVIL II baseline
• The open-architecture track reached 3.211, well below both backprop references
• The method injects Gaussian noise into per-token activations instead of backpropagating
• Experiments used a 37.7M-parameter model trained on the FineWeb dataset