Take the AI Away, and Solo Scores Drop: Three Experiments With 1,222 Participants
Across three experiments with 1,222 US participants, people who had just used AI scored lower on problems they then solved alone than people who never used it. The researchers say they could not determine the effects of longer use or how long the impact lasts.
On October 9, UC Berkeley highlighted a study that looked at how people perform when the AI help they had been leaning on is suddenly taken away. Across all three experiments, which covered fraction arithmetic and reading comprehension, the group that had used AI scored lower afterward than the group that had not. The work is a revision of a preprint released in April, and it was presented at the Conference on Language Modeling (COLM) in October.
What happened when the AI disappeared
The researchers recruited 1,222 online participants in the United States across three experiments. Participants worked through fraction arithmetic problems or SAT-style reading comprehension questions. Some of them were able to ask GPT-5, running in a side panel, for help for roughly 10 to 15 minutes. The researchers then removed the AI without warning and measured accuracy and skip rates on the final three questions.
The experiments focused on what happens when people have to solve problems on their own immediately after receiving help. Participants were randomly assigned to groups, and the researchers compared outcomes across two different kinds of tasks: calculation and reading.
How far did accuracy fall
In the first fraction experiment, the group that had used AI scored 16 percentage points lower than the group that had not. Accuracy on the final three questions was 57 percent and 73 percent, respectively. Skip rates also differed: 20 percent for the AI group versus 11 percent for the control group.
The second fraction experiment pointed in the same direction, with accuracy at 71 percent for the AI group and 77 percent for the control group. Skip rates, however, were 10 percent and 7 percent, a difference that was not statistically significant. In the reading comprehension experiment, accuracy was 76 percent versus 89 percent, and skip rates were 8 percent versus 1 percent. Accuracy was lower in all three experiments, but the difference in skipping behavior varied from one experiment to the next.
What happened to people who just took the AI’s answers
The researchers also examined what kind of help participants asked the AI for. In the second experiment, 61 percent of those with AI access said they mainly used it to get answers. Another 27 percent said they used it for hints or explanations. The group that used AI to get answers went on to post lower accuracy and a higher skip rate. The researchers noted that this analysis cannot establish whether the manner of use was what produced the difference.
The team said it was unable to determine whether the effect compounds with longer AI use, or whether any impact persists hours or days later. The experiments were run with participants in the United States on English-language tasks.
For readers who use AI for studying or for work, the study poses a single question: can you still solve the problem on your own once the help is gone? The things worth watching next are the difference between taking the answer and asking for a hint, how long the effect lasts, and whether similar results show up in Korean-language learning settings.