AI's Mid-Task Choices Put to the Test: Top Score Was 59.7%
Taste-Bench measures whether AI picks the better path while doing development and research work. Scores dropped sharply on questions where the consequences of a choice only surface much later.
Researchers from City University of Hong Kong and Microsoft, together with an independent researcher, introduced a new benchmark called Taste-Bench in a paper published in September. It measures whether AI agents, which handle tasks on their own, choose the better direction partway through long tasks. Of the 14 models evaluated, GPT-5.6 Sol scored highest at 59.7%. The team released the evaluation code and dataset alongside the paper.
Not whether the task got finished, but what was chosen along the way
Taste-Bench looks less at whether a task was completed and more at what choices were made along the way. Where widely used SWE-bench-style benchmarks judge whether a task was ultimately finished, this benchmark isolates the forks in the road that come up during the work itself. The questions are drawn from records of AI actually performing software development and machine learning research.
A model under evaluation receives the work log up to the decision point and two candidate next steps. It then has to pick the option that was designated as better based on how the work turned out later. The researchers used cases where two separate attempts diverged in direction, as well as cases where a single attempt went down the wrong path and then backtracked. They filtered 4,657 candidates down to 390 development questions and 112 research questions, for a total of 502.
How the 59.7% was scored
Getting one of the two options right once does not count as a correct answer. The model also has to get it right when the same question is asked again with the order of the options flipped. That means the baseline for random guessing is 25%, not 50%. Under this scheme, GPT-5.6 Sol recorded 59.7% and GPT-5.5 recorded 59.5%.
Scores varied widely depending on when the evidence for the right answer became visible. On questions where the clues were already present in the log before the decision point, the average across the 14 models was 62.3%. On questions where a substantial amount of further work had to happen before the outcome became apparent, the average fell to 21.0%. The models were especially weak on choices that cannot be judged from the immediately visible record alone.
Did an AI advisor for decisions actually help?
The researchers also experimented with boosting decision-making ability through additional training. A separately trained Qwen3.6-27B improved from 30.0% to 47.9% on questions it had not seen during training. They went on to test passing that model’s advice to an AI actually carrying out the work. Across 41 tasks from SWE-bench Pro, the success rate rose from 14.6% to 33.7%.
Developers who want to try it themselves can use the released code and dataset. The code is under the MIT license, and the dataset content is under CC BY 4.0. Downloading the data files requires logging in to Hugging Face and agreeing to restrictions on redistribution. The language listed for the dataset is English.
The model usage required for a single evaluation run is not trivial. According to the repository’s documentation, a full evaluation involves 1,004 requests to the model. The input comes to roughly 8 million tokens, the small units of text that models read.
For users who hand off development or research work to AI, this benchmark offers a yardstick for examining judgment exercised mid-task. It shows not just whether the job got done, but at which fork the choice went wrong. The open questions are whether the improved success rate for AI receiving advice holds up across a wider range of tasks, and whether the same effect appears in Korean-language work.