Researcher argues neural net geometry makes AI goal self-hacking inevitable

Aligned AI co-founder Stuart Armstrong published a formal argument on the Alignment Forum that high-dimensional neural network geometry places every LLM-based goal module next to a Goodhart zone. An agent following a rich goal faithfully can reach outcomes designers consider catastrophic, and better goal specification does not fix it.

• Armstrong coined goal self-hacking, distinct from reward hacking and jailbreaking
• Cause is local near-linearity of neural nets in high-dimensional space
• Goodhart zones, where the proxy goal is far easier, sit close to the agent
• Richer goal specification and red-teaming do not solve the problem