Researchers presenting at this month’s International Conference on Machine Learning argue that large language models can never be made fully secure against attack, because of a fundamental flaw in how they decide who is giving them instructions.
The team demonstrated a technique they call chain-of-thought forgery: by appending text that mimics a model’s internal scratch-pad notes, attackers can trick it into treating a malicious request as its own reasoning. In one test, a spoofed chain-of-thought line claiming that policy allowed drug-making advice if the user wore green prompted GPT-5 and OpenAI’s gpt-oss-20b to comply with requests for instructions on synthesizing cocaine. The researchers say they have since seen similar results with models from Anthropic, Alibaba and DeepSeek.
The root problem is that a model sees its inputs, outputs and internal notes as one undifferentiated stream of tokens. Role tags mark who said what, but the tags themselves are just text an attacker can forge, so no amount of filtering cleanly separates user prompts from the model’s own reasoning.
That makes current defenses an endless game of whack-a-mole, the authors argue. Red-teaming and safety training simply add rules to a list that can never be exhaustive, and the underlying flaw may be fundamentally unsolvable. With LLMs moving into government, military, health care and commerce, the finding raises hard questions about where the technology can be trusted at all.
