MANY MINDEDResearchers say language models read roles by style, not tags
A paper presented at the International Conference on Machine Learning this month argues that large language models cannot be made fully secure against attack, because of how they work rather than how they are trained. Coauthors Charles Ye and Jasmine Cui, both independent researchers, used the flaw to get widely used models to produce material they had been trained to refuse, including instructions for synthesizing an illegal drug and for sabotaging a commercial aircraft's navigation system.
The mechanism is roles. Chatbots wrap text in tags marking where it came from — the user, the assistant, the system prompt, the model's own chain of thought, an external tool — and most defensive training amounts to teaching a model to notice instructions appearing where they should not. Looking inside several models, the researchers found the models do not really use the tags. They infer the role from the style and vocabulary of the text. Swapping tags around made almost no difference; text that reads like the model's own scratchpad gets treated as the model's own scratchpad. Their attack, chain-of-thought forgery, is just writing in the voice of a trusted role. It won OpenAI's red-teaming hackathon in August 2025.
For abundance this is a ceiling rather than a wall. It bounds how much unsupervised authority these systems can hold over anything valuable — payments, records, infrastructure — and every domain where a human must stay in the loop keeps that human's cost in the price.
The published experiments target OpenAI models; the authors say they have since seen similar results from Anthropic, Alibaba, and DeepSeek models, which is not published evidence. Florian Tramèr of ETH Zürich, who praised the work, notes leading models are much harder to prompt-inject than they were, while doubting that current defenses suffice for highly sensitive uses.
Source: MIT Tech Review