Google Experiment: AI Lifts Document Quality, but Junior Attorneys' Skills Stay Flat

Google published research on October 7 (local time) examining how AI changes both job performance and underlying skill. Over three months, patent attorneys who used AI produced better drafting work regardless of how experienced they were. But on a task where they had to find and fix errors without AI, only the experienced attorneys scored higher than the comparison group. Junior attorneys showed no average difference from the comparison group, with both strong scores and very poor scores becoming more common.

How much did AI improve document quality?

The researchers recruited 133 attorneys from 11 intellectual property law firms that work with Google on a non-exclusive basis. At each firm, roughly two-thirds were randomly selected to use a patent drafting AI tool from Google Labs that had not yet been made public. The rest received some training on AI use but were asked to wait until after the experiment to use the tool itself. That tool is now part of Gemini Notebook.

Participants drafted patent documents from fictional invention materials at 10 days and 90 days after the experiment began. The AI group was encouraged to use either this tool or other AI systems. Outside patent experts handled the scoring, evaluating five dimensions: the enforceability of the legal rights claimed, accuracy, scope of claims, completeness, and clarity.

The AI group scored higher on both rounds. The researchers explained that converting the gap into a ranking within the peer group amounts to a gain of 10 percentile points at day 10 and 11 percentile points at day 90. Looking at the score distribution, low scores declined and mid-range scores increased. The frequency of truly outstanding scores, on the other hand, did not change. Junior attorneys also saved time: on the day 10 task, they finished 18 minutes faster than the comparison group’s average of 124 minutes.

What happened when AI was taken away?

At day 90, the researchers assigned a separate task in which AI was off-limits. Participants had to personally correct a fictional patent document seeded with errors in both substance and wording. Reviewing a patent requires the judgment to catch errors that overstate an invention’s novelty or the breadth of its claims in ways that can undermine legal force. The design was meant to measure, separately, the quality of AI-assisted output and the skill a person can exercise on their own.

Source: Google (press release)

On this task, the overall score of the group that had been using AI was 0.32 standard deviations higher than the comparison group. The researchers noted, however, that the entire gap came from the experienced attorneys. Their score difference was 0.45 standard deviations, while junior attorneys showed no clear average improvement. Within the junior group, mid-range scores declined while both strong scores and very poor scores increased.

Junior and experienced attorneys edited differently

Differences also showed up in how they went about revising. Junior attorneys, whether or not they had used AI, worked through the document in order from the beginning and spent their time polishing low-priority wording. Even when they spotted a serious flaw, they often left a comment flagging it rather than fixing it themselves. Experienced attorneys who had used AI, by contrast, focused on restructuring the core claims. In follow-up interviews, they said they treated AI output not as a finished product but as a tool for stress-testing their reasoning.

Source: Google (press release)

The researchers also flagged limitations. Participants were limited to patent attorneys, and the observation window was only three months, short compared with the years it takes to build expertise. They added that because AI model capability and user familiarity have both shifted in the meantime, running the same experiment today could yield different results.

The researchers suggested that pre-existing expertise may be the link that converts AI experience into genuine skill growth. Whether junior workers move beyond producing good output to catching critical errors on their own is a question that needs longer observation. The message of this study is that when assessing the payoff from adopting AI, the quality of the output and the judgment of the employee need to be measured separately.