HotKV cut Claude token use by 70% in Banking77 benchmark

Semantic cache HotKV reduced model calls, tokens and API cost by about 70% across four Claude models on 3,000 Banking77 support requests. Roughly 39% of answers came from exact matches and 31% from semantic matches, though at a 0.81 similarity threshold cached answers fit the exact question 5.8–10.7 points less often than fresh ones.

• Calls fell from 12,000 to 3,588, tokens by 70.1% and cost by 69.6%
• About 39% of requests were served by exact cache, another 31% by semantic
• Median response time dropped from 1.4–2.9 s to 125–128 ms
• At a 0.81 threshold, 96.7–96.8% of semantic hits matched the intent