Microsoft Trains AI Tutors on Simulated Students, Reaching 90.5% Accuracy

Microsoft Research introduced StudentSim on October 12 local time, a research system for evaluating AI tutors. It works by imitating the answers a student gives and how those answers change after an explanation. The team also released StudentSimEval, a benchmark that measures this across three domains: chess, English writing as a second language, and math.

StudentSim is a research tool meant for the development stage, before a tutor ever reaches actual learners. The researchers said they started from a practical problem: experiments that collect responses directly from students take a great deal of time and money. That makes it hard to compare different tutoring approaches quickly while a system is still being built.

Why simply casting an AI as a student falls short

Simply instructing an AI to “answer like a student” does not reliably reproduce a real learner’s level, according to the researchers. A model can talk like a young student while still producing answers that draw on knowledge well beyond that level. When a simulated student is too capable, a tutor’s explanations look more effective than they really are. Existing models that analyze learning logs can predict a student’s state, the researchers added, but they struggle to capture how a student would respond to an explanation given in words.

StudentSim looks at two things together. First, it checks how closely the system reproduces the answers a specific student actually wrote on problems that were not used in training. Then it examines how those answers change after the student reads the tutor’s guidance. The design aims to capture not only a student’s baseline ability and mistakes, but also how they shift when given help.

Capturing what makes each student different

Training happens in two stages. First, the model learns common answer patterns, errors, and ways of revising an answer after an explanation, drawn from the logs of many students studying the same subject. Then it is further trained on one student’s records to create a simulated learner tailored to that person. The researchers said they used only logs released for research purposes, and that no personally identifying information was used at any stage.

Image: Microsoft Research

The responses being imitated differ by subject. In chess, the system predicts the student’s next move and the move they would switch to after receiving guidance. In English writing, it handles text containing a student’s characteristic errors along with the revisions that follow feedback. In math, it predicts multiple-choice answers and how those answers change after an explanation. The comparison models were given the same per-student training and evaluation records.

Did tutors trained on simulated students actually improve?

The researchers reported that StudentSim outperformed the main comparison models in all three subjects. Higher scores mean responses closer to those of the real student. On reproducing students’ original responses in chess, StudentSim scored 0.5150 against 0.2316 for GPT-5.4. On reproducing responses after guidance, the scores were 0.9067 and 0.7186 respectively. Maia2, a model that predicts human chess moves, scored 0.4535 and 0.2721 in the same order.

Image: Microsoft Research

The team also ran an experiment that trained a chess tutor using the simulated students’ reactions. The tutor was rewarded when a simulated student who had read its explanation produced a better move than before. Chess experts graded the results without knowing how each tutor had been trained, and the tutor trained with StudentSim scored 90.5% on accuracy. A tutor that had not gone through the same reward training scored 75.7%, and one that used GPT-5.4 as the simulated student scored 71.6%. The StudentSim-trained tutor also earned higher marks on guidance quality and personalization.

The paper and the code used for training and evaluation have been released publicly. The researchers stressed that studies with human participants are still needed to judge actual learning gains, safety, usability, and classroom fit. The point to watch going forward is whether explanations that worked on simulated students also translate into better outcomes for real ones.