Grading Work Records to Train AI: Tasks Solved Every Time Jump From 38 to 107

Microsoft and Hugging Face released ThinkingBox-Training on GitHub on October 22 (local time), a method that turns the results of an AI’s finished work into training signal. The two companies graded the records left behind in a database and used those scores to further train Qwen3.8-27B. Average success rate rose from 51.70% to 60.78%, a gain of 9.08 percentage points.

Why not trust the AI when it says “done”?

ThinkingBox waits until an agent, an AI that handles tasks on its own, has finished a job, then compares the database records against the expected outcome. In an earlier example the researchers described, an AI marked a customer inquiry as resolved while the related shipping exception was still left unresolved. The yardstick is not whether the tool calls looked plausible, but whether the work records are actually correct.

This time, those grades were fed into reinforcement learning, a method that improves a model by rewarding successful behavior. The researchers trained Qwen3.8-27B for 50 steps on 187 tasks chosen separately from the evaluation set. They picked the practice tasks by first checking how likely the model was to succeed on each one before this additional training, and left that list unchanged throughout training. Tasks where success and failure are mixed, they explained, offer more to learn from than tasks the model always passes or always fails.

How many tasks succeeded all 20 times?

Tasks that succeeded on every attempt went from 38 before training to 107 after. The researchers ran all 507 tasks in ThinkingBox-Bench, none of which were used in training, 20 times each, for 10,140 attempts in total. The share of tasks that succeeded all 20 times climbed from 7.5% to 21.1%. The share of tasks that succeeded at least once out of 20, on the other hand, dipped slightly, the researchers reported.

Image: huggingface.co

In a hotel reservation change scenario, the model followed approval rules far more often. Company policy required manager approval to downgrade a room class, but before training the model changed the booking and issued a refund without approval in 15 of 20 attempts. After training, it preserved the reservation and payment status in all 20. In 19 of those attempts, every expected record matched as well.

A high average score did not necessarily mean consistent repeat success. In the researchers’ comparison, GPT-5.6 Sol posted an average success rate of 61.91%, higher than the 60.78% of the trained Qwen3.8-27B. Yet the number of tasks solved all 20 times was 82 and 107, respectively. The researchers stressed that when choosing a model to handle real work records, average success rate and repeat reliability should be weighed together.

Will it hold up in real work?

The researchers said the estimated total evaluation cost divided by the number of successful attempts fell from $0.177 to $0.139. That is a comparison figure that folds in the cost of failed attempts, not an actual bill. It also excludes the cost of training and of selecting the practice tasks. Training used 24 Nvidia H100s.

Image: huggingface.co

The evaluation tasks were built from business domains and databases the model had not seen during training. The researchers did, however, acknowledge a limitation: training and evaluation share the same execution framework, the same grading method, and some of the same tools. Since this is a case of lifting an AI’s repeat success rate in work where record accuracy matters, such as reservations and payments, the next hurdle is whether the gains carry over to settings outside that shared foundation.