Whether the job got done is the only question most agent benchmarks ask. A paper from Microsoft and City University of Hong Kong says the answer hides the part that decides success.
Long-running work means constant forks — a hypothesis to test, a codebase to extend, an experiment to queue up. A weak pick looks harmless in the moment, and only charges its bill once the budget is nearly spent.
Taste is the authors’ name for the skill. Their benchmark, Taste-Bench, repairs the measurement problem by grading after the fact rather than with human raters. Agents attempt a task more than once, the runs split at identifiable points, and how each attempt ends supplies the label for which branch was better.
Those split points are what they call decision forks. A run is frozen there, everything after it is hidden, and a model has to choose between two directions. The questions number 502, sourced from SWE-bench Pro, METR’s MALT transcripts and a large pool of scored runs across coding and research. Every question was asked twice with the options swapped, so only consistently correct answers counted. Pure guessing returns 25 percent.
Across 14 models spanning Mistral, MiniMax, GLM, DeepSeek, Grok, GPT and Claude families, GPT-5.6 Sol led at 59.7 percent and GPT-5.5 followed at 59.5, with every other model far behind.
Position of the deciding clue proved decisive. Where it sat inside the preceding steps, average accuracy reached 62.3 percent. Where it surfaced only after extensive later work, accuracy fell to 21 percent.
More thinking did not help. Scores barely moved when reasoning effort rose, and the longest deliberations landed on the questions models missed most often. Training a student model on a teacher’s judgments also lifted agent success on held-out SWE-bench Pro tasks from 14.6 to 33.7 percent.
