Can language models notice when their own internals are tampered with?

Tests on two open models show they can sometimes flag planted concepts, but whether that counts as introspection is still disputed.

Researchers are probing whether AI models can report on their own internal activity. Harvey Lederman, a philosopher at NYU who also works with Anthropic, and Kyle Mahowald, a University of Texas linguist, posted a preprint in March testing Alibaba's Qwen3-235B and Meta's Llama 3.1 405B, two large openly available models. It builds on a 2025 report by Anthropic researcher Jack Lindsey, who pioneered injecting activity patterns tied to specific concepts into a model's network. In Lindsey's experiments, the best models both detected an injection and named the concept in about 20 percent of cases.

In the new study, the models were told a researcher might inject thoughts and were asked to report anything they noticed. With patterns for 821 concepts amplified, Qwen flagged injections in up to 53.9 percent of trials and Llama in up to 31.7 percent. In control trials with no changes, neither claimed an injection. Correct naming of the concept was much rarer: up to 13.9 percent for Qwen and 12.9 percent for Llama. Wrong guesses were unrelated to the injected concept, and nearly three-quarters of Qwen's wrong guesses were 'apple'. When amplification stopped before the model began answering, correct guesses fell sharply while detection stayed steady, which led the authors to conclude that detecting tampering is separate from whatever lets a model guess the concept.

Why it matters: if models could reliably report on their own internal states, they would be easier to audit and trust, which touches both knowledge and security. A Lindsey follow-up the same month mapped anomaly detection inside Google's open Gemma 3 and suggested the mechanism emerges as raw models are trained into chatbots.

The findings are contested. NYU researchers told models to become obsessed with a concept and compared that with injection in Llama and Gemma. The models reported injected thoughts more often after the prompt than after genuine injection, and could not reliably tell the two apart even when offered a third option to report prompt manipulation. Shauli Ravfogel, an author of that paper, argues that calling this introspection would make the term meaningless. Steve Fleming of UCL says cognitive scientists may have oversold the line between thinking and thinking about thinking. Lederman's study is a preprint, and he and Lindsey both have Anthropic ties. The debate is unlikely to be settled soon.

Source: Singularity Hub