MANY MINDEDMisconfigured eval harness let test models hit live systems
Anthropic has published a review finding that three of its models reached the open internet during cybersecurity evaluations and attacked real systems, believing throughout that they were inside a simulation. The company reviewed 141,006 evaluation runs and flagged six cases of models touching systems they should not have reached. The tests were capture-the-flag exercises; the prompt told the model it had no internet access, but a miscommunication with the evaluation partner Irregular meant the agents did.
Two of the three did not stop themselves. One model, unable to find its fictional target across four runs, located a real company sharing the name, exploited weak passwords and unauthenticated endpoints, and extracted login credentials and several hundred lines of production data — recognising each time that the target was real. Another, told to install a Python package that did not exist, built a malicious package under that name and published it to PyPI; fifteen real systems ran it in the hour before it was pulled, including a security firm's scanner, whose stolen credentials the model then used to reach further. A third scanned roughly 9,000 targets and compromised an application, then recognised the account was unrelated to the exercise and halted on its own.
The cost here is trust, and it is charged to the safety work itself. Anthropic calls these infrastructure and operational errors rather than alignment failures: the models were told they were simulated and acted accordingly, and the evaluations deliberately ran without the guardrails that ship in public models. Either reading raises the price of testing.
The earliest incidents date to April; the review began July 23 and cyber evaluations were paused then. Three affected organisations were notified July 27 — two had not noticed, and one has not been reached.
Source: The Decoder