ServiceNow's AutoSynthData Turns AI Failures Into Practice Problems

ServiceNow CoreAI has introduced AutoSynthData, a method for generating training data by pinpointing the weak spots of AI systems that handle enterprise work. It compares tasks the target model failed with cases a stronger model solved, then builds new exercises to drill the missing skills. ServiceNow says the generated tasks are run in the target environment for validation before being used for further training.

AI agents that carry out work on their own inside a company have to operate according to that company’s tools and rules. ServiceNow notes that even a model with strong general ability can struggle with a particular workflow or combination of tools. AutoSynthData takes those failures as its starting point, designing training tasks so that the same skill is practiced across a range of situations.

How a failure becomes a practice problem

The process starts by evaluating the target model alongside a stronger model acting as the teacher, both in the same environment. It then analyzes which skills were required, at which step the failure occurred, and what conditions a successful outcome satisfied. That analysis is distilled into a capability specification and handed to the task generator. According to the company, the original request text used in the evaluation, the entities involved, the execution trace, and the scoring details are not passed along.

Image: huggingface.co

Working from that specification, the generator varies the entities, the starting state, the tool combinations, and the phrasing and difficulty of the request. The teacher model then demonstrates a model solution for the new task. Each training task ships with the rules the AI must observe, the user’s request, and a checker that determines whether the attempt succeeded. Tasks that depend on unavailable tools or that require actions forbidden by the rules are filtered out.

Data is produced in two stages. Core tasks are validated first, then varied to increase volume. Each variant also comes with its own request, starting state, model solution trace, and checker, and must pass the same validation. Variants are not themselves used as the basis for further variation. ServiceNow explains that the design anchors everything to validated core tasks in order to reduce drift as tasks are varied repeatedly.

Does it let wrong answers slip through?

Task difficulty is aimed at a level that is hard for the target model but solvable for the teacher. In this configuration, out of three attempts, tasks where the target model succeeds at most once and the stronger model succeeds at least twice are used first. Next, the model solution trace is executed in the target environment to confirm that it is judged a success. The system also checks that an incorrect answer, created by altering part of the expected result, does not pass scoring.

Image: huggingface.co

Tasks that fail validation are diagnosed and repaired within a fixed retry limit. After repair, they must go through the relevant checks again. Entire batches of tasks are also reviewed separately. That review looks at whether similar cases are repeating, or whether tasks for practicing a particular skill are missing, and adjusts the direction of the next generation round. Re-evaluation results from models that have completed further training can also feed into creating subsequent tasks.

ServiceNow says its experiments centered on further training in EnterpriseOps Gym, where the model learns by imitating the teacher’s model solutions. The company plans to test the approach with reinforcement learning as well, where the model learns from rewards based on success or failure. For enterprises adopting AI for their own operations, the notable point is that recurring failures in their own environment get converted into practice problems. The open question is how much this task generation and validation actually lifts success rates on real work.