Haiku 5.5 vs 4.5: 100-prompt test shows far less repeatable answers
A developer ran the same 100 fixed prompts through Claude Haiku 4.5 and 5.5 twice a day for 11 days. Haiku 5.5 returned an identical answer on only 41–49 of 100 questions versus 95–100 for 4.5, rejects temperature and top_k, and adds a thinking block to a third of responses.
- Haiku 5.5 repeats answers on 41–49 of 100 prompts, 4.5 on 95–100
- Haiku 5.5 returns a 400 error for temperature other than 1 and for top_k
- 33% of 5.5 responses start with a thinking block; 2.2% hit the 1024-token limit
- Haiku 5.5 costs a tenth of 4.5 per token but uses more output tokens
Read next
AI