Georgia Tech, Nvidia and Stanford's BOOST speeds LLM inference by 31%

Researchers at Georgia Tech, Nvidia Research and Stanford published BOOST, the first runtime system providing concurrent access to host memory and HBM for LLM inference without kernel changes. On a Grace Hopper system, BOOST improves throughput by 31% on average and Time-per-Output-Token by 4.3% at iso-batch size.

• BOOST enables concurrent host memory and HBM access without kernel changes
• Average throughput gain of 31% over HBM-only serving
• TPOT improves 4.3%, while prefetching degrades it by 6%
• Integrated into vLLM and evaluated on a Grace Hopper system