A complaint familiar to anyone relying on a search-backed assistant now has figures behind it: the links often bear no relation to the answer, and nobody opened them. The work comes out of Imperial College London, where three researchers in the Department of Computing produced it.
Understanding Why Search-Augmented LLMs Fail to Use Available Evidence — that is the title of their paper, which opens by noting that models answer off search snippets rather than opening source pages. One proprietary system, OpenAI’s GPT-5.6 Luna, joined six open-weight configurations in tests against hundreds of facts that shifted after each model’s training cutoff.
Answering from the blurb
Three things were tracked over eight configurations: how accurate answers were, how often models leaned on snippets while leaving source pages shut, and citations that ran against what the model itself claimed. In certain runs, searches concluded with zero pages fetched more than fifty percent of the time.
Worse outcomes trailed that shortcut. Compared with models that read what they retrieved, those answering from snippets generated far more outdated answers and misleading attributions.
What the authors describe is a gap between evidence being available and being used well. A layer-wise analysis surfaced cases where retrieved evidence shaped internal predictions yet the final answer still leaned on older information.
The old facts keep winning
Failure rates fell substantially when the correct evidence was supplied directly. Eliminated they were not. Even with the newer fact present in the context window, some systems clung to their stale answer.
Economics account for the behavior. Page downloads and reads consume bandwidth and latency, and the page might be gone regardless. Cheaper are snippets. By showing that the cheap route is not just less thorough but measurably less accurate, the paper matters for any workflow treating a citation as proof.
