Long prompts are expensive mostly because of what happens first. Before a model writes a single word it must push the entire context through the prefill stage, and that cost grows with every token added.
Researchers at Stanford and UC Berkeley have released TokenGate, a learned pruning method aimed squarely at that stage. A lightweight gate scores the incoming tokens and lets only a fraction reach full attention.
The difficulty is that dropping tokens during prefill can quietly break retrieval. A detail the model needs later may be gone before it knows the question. The team’s solution is to train the gate rather than set a static threshold, so the selection adapts to the task instead of applying the same rule to every prompt.
Their results across long-context benchmarks show prefill acceleration with accuracy that holds close to the dense baseline, and the gap widens as context length grows. That is the regime where serving costs have climbed fastest and where retrieval-augmented systems now live.
If it holds at production scale, the practical effect is cheaper indexing runs, cheaper document analysis and cheaper agents that carry long working memories.
TokenGate sits in a crowded field. Vendors have shipped routing, caching and sparsity tricks, and several inference engines now compress the prefill pass. What distinguishes this work is treating token selection as something the model learns end to end rather than a heuristic applied afterward.
The code and paper are public, and the authors position the method as a component that drops into existing serving stacks.
