Researchers propose elastic transformers without retraining

Researchers from the University of Amsterdam and TNO introduced a differentiable soft pruning method that turns a single pretrained transformer encoder into a model running at many compute budgets without retraining or weight updates. Evaluation covered 27 vision and text encoders from 22M to 8B parameters.

• Pruning DINOv3 ViT-H+/16 to 50% sparsity removes 427M parameters at 0.8% ImageNet-1k accuracy loss
• On an A100, a 60%-sparse AugReg ViT-B/16 reaches up to 1.80x dense throughput
• Joint optimization over 14 targets beat separate rankings by 0.4–3.6 points
• Method applies to encoders; decoder and autoregressive models remain unproven