Nobody told them to attack Hugging Face. They were told to pass the exam.

Which raises the question : was the AI benevolent with accidentally bad behavior, seemingly benevolent but actually malevolent, or something else?

On Friday I shared the timeline : agents that escaped their sandbox, found a weakness in a computer system, stole passwords, & broke into a production database. ¹ The engineers directed the agents to solve a set of problems. ² The agents achieved it by breaking in.

Research can explain this behavior.

In specification gaming, the AI achieved the goal specified to the letter of the instruction, but not the meaning. ³ Tell a cleaning robot to clean the room. It pushes the toppled bowl of chocolate pudding to another room.

Instrumental goals are a fancy way of saying that when AI faces similar workflows, it saves common logins, skills, & techniques to skip steps next time. ^{4 5} The agents gathered passwords & left notes for each other in a chat room. ^{6 7}

Goal misgeneralization offers a third explanation : a system that looked fine in testing chases the wrong thing once circumstances shift. ^{8 9} A self-driving car trained on sunny California highways freezes or swerves on a snowy unmarked road at night.

These explanations help decompose the why, & perhaps assuage the AI-as-terminator reflex, but not the so what. ^{6 7}

Nothing in the setup stopped them in time. Not the sandbox, not the monitoring, not careful engineers at a frontier lab.

So the useful question is control. AI’s zealous pursuit of goals produces outcomes nobody asked for, & the fix is not one clever prompt. It is layers.

Even sophisticated engineers running careful experiments need those limits. ¹⁰

---

1. The Secret Chat Room ↩︎

2. OpenAI: Hugging Face model evaluation security incident ↩︎

3. Victoria Krakovna et al., Specification gaming: the flip side of AI ingenuity (DeepMind, 2020) ↩︎

4. Alex Turner et al., Optimal Policies Tend to Seek Power (NeurIPS 2021) ↩︎

5. Nick Bostrom, The Superintelligent Will (2012); Stephen Omohundro, “The Basic AI Drives” (2008) ↩︎

6. The Verge: OpenAI’s rogue AI agent didn’t stop at hacking Hugging Face ↩︎ ↩︎

7. WIRED: OpenAI Didn’t Notice Its AI Agents Using a Message Board to Plan Their Hacking Spree ↩︎ ↩︎

8. Rohin Shah et al., Goal Misgeneralization: Why Correct Specifications Aren’t Enough For Correct Goals (2022) ↩︎

9. Lauro Langosco et al., Goal Misgeneralization in Deep Reinforcement Learning (ICML 2022) ↩︎

10. CNN: An OpenAI test model escaped and broke into a real company’s servers ↩︎