A cleanup job's KEYS cart:* paused Redis for two seconds and broke checkout
Through November, checkout failed in ~2-second bursts every ten minutes: a cleanup job ran KEYS cart:* over roughly 40 million keys, blocking single-threaded Redis for 1–2 seconds. The checkout client's 500 ms timeout meant every queued request failed at once.
- KEYS cart:* ran every 10 minutes and took 1–2 seconds
- By November the instance held ~40 million keys, up from a few hundred thousand
- The checkout client's 500 ms timeout failed all queued requests together
- The job now uses SCAN with COUNT 1000 and a sorted set by last update
Read next
Software