IBM Lifted GPT-4.1's Customer Service Success Rate by 10 Percentage Points Just by Adding Guidelines

IBM researchers have improved a customer service AI’s performance simply by attaching task guidelines distilled from its own past experience. The results, from an experiment called ALTK-Evolve, were released on October 20. In retail-domain testing with GPT-4.1, the share of tasks the agent completed successfully on all four attempts rose from 57.5% to 67.5%. The gain came without any additional model training, achieved purely by augmenting the prompt, the set of instructions given to the AI.

How it remembers past mistakes

ALTK-Evolve analyzes records of how the AI handled training tasks and turns them into short behavioral rules. Verify the customer’s identity first, for example, or summarize the proposed changes and get the customer’s consent before modifying an order. Another guideline that emerged: itemize the refund amount and explain the refund timeline up front. The researchers used the same GPT-4.1 both for extracting the guidelines and for handling customer service, and said no human-annotated ground truth labels were needed for training.

The extracted rules are appended to the end of the base instructions the AI reads at the start of every conversation. The researchers left existing business policies, tools, and the agent’s execution architecture untouched, adding only the guidelines so they could isolate the effect.

How much did retail success rates rise?

The experiment used τ²-bench, a customer service benchmark built by Sierra. In it, the AI converses with another AI playing the customer and handles jobs such as looking up orders or processing refunds. Scoring does not reward merely plausible-sounding answers: a run counts as a success only if the database ends up in the state the task expected. The researchers ran four trials per task across the retail and airline domains.

Image: huggingface.co

On tasks held out from training, the average retail success rate climbed from 78.1% to 82.5%. The share of tasks solved on all four attempts (Pass^4) rose by 10 percentage points. In the airline domain, applying the full set of guidelines lifted the average success rate from 57.5% to 65.0%. Pass^4, by contrast, peaked when only the 14 guidelines that recurred across multiple training tasks were included, rising from 35.0% to 45.0%.

The researchers also reported a retail score of 67.5% across the full task set, higher than every entry on the October 12 leaderboard. That figure combines both the training and test tasks. It is 6.1 percentage points above the 61.4% posted by RAFT-30B-A3B, which was further trained on synthetic retail data. In the airline domain, swapping the model for GPT-5.5 and re-extracting the guidelines pushed Pass^4 on the held-out test tasks from 55.0% to 70.0%.

Shorter conversations, higher costs

Better scores did not come at the price of longer conversations. Comparing matched runs that succeeded both with and without guidelines, the retail agent using the full guideline set averaged 10.16 response turns, down from 10.59. In the airline domain, the average number of response turns also fell, from 9.15 to 8.82.

Image: huggingface.co

The tradeoff is cost: the instructions the AI must read on every turn grow longer. In retail, adding 292 guidelines raised the cost per conversation from roughly $0.084 to roughly $0.139. That figure combines the usage fees for both the customer service agent and the AI playing the customer. The researchers noted that caching and reusing the repeated portions of the prompt could absorb much of that increase in real deployments.

The experiment shows that organizing operational experience into explicit guidelines is a viable option for improving customer service AI. Anyone deploying it in practice will need to tune how many guidelines each domain gets. The open questions for further validation: whether the same gains appear in Korean-language support conversations, and how success rates and costs shift when the system faces real customers.