Benchmark tests whether LLMs invent Japanese law articles

A developer checked Japanese law article citations from three local LLMs (1.8B–8B parameters) against the official e-Gov registry of 6,913 articles. llm-jp-3-1.8b invented 4.57% of articles in Japanese answers versus 1.09% in English, Qwen2.5-7B 1.17% versus 0%, and Swallow-8B none in either language.

• llm-jp-3-1.8b: 63 of 1,379 invented citations in Japanese, 7 of 642 in English
• Qwen2.5-7B: 7 of 597 in Japanese, 0 of 567 in English
• Swallow-8B invented no articles in either language
• The e-Gov registry covers 6,913 articles, including deleted placeholders