DeepSeek's Engram adds a lookup table to transformers, beats MoE twin

A DeepSeek-linked team published "Conditional Memory via Scalable Lookup" (arXiv:2601.07372), introducing Engram, a deterministic N-gram lookup module for transformers. Engram-27B (26.7B parameters, 3.8B active per token) beats an iso-parameter, iso-FLOPs MoE baseline: BBH +5.0, HumanEval +3.0, and 32K NIAH rising from 84.2 to 97.0. Code is open on GitHub.

• Engram-27B: 26.7B parameters, 3.8B active per token, same budget as the MoE baseline
• BBH rose from 50.9 to 55.9; HumanEval from 37.8 to 40.8
• Multi-Query NIAH at 32K: 84.2 → 97.0 (+12.8)
• Matches baseline perplexity at 82% of pretraining FLOPs (41K vs 50K steps)