Swapping a Coding AI's Harness Can Cost Up to 3 Times as Much, With Similar Performance
In a study that attached different harnesses to the same model, Claude Code and mini-SWE-agent resolved tasks at virtually identical rates. The cost gap came mostly from the length of instructions resent at every turn and the number of steps taken.
How much do a coding AI’s results and costs shift when you change the harness around it? On October 3, Yangze Liu and Zhongyi Han, researchers at Shandong University, released a paper comparing those two variables. Even with the same underlying model, the cost per task varied by as much as 3x. Yet across an experiment covering 447 tasks, the gap in resolution rate between Claude Code and mini-SWE-agent stayed within 5 percentage points for both of the models they ran. The researchers concluded that the two tools perform at effectively the same level.
Same model, different harness: what was compared
What the researchers compared was not the model but the harness that wraps the model and puts it to work. That includes the system prompt handed to the agent (an AI that handles coding work on its own), the list of tools it can call, and how the back-and-forth conversation is managed. The team connected the same model to Claude Code, mini-SWE-agent, and OpenCode. For the test set they used SWE-bench Verified, which measures the ability to solve real coding tasks.
Their comparison also accounted for the score variation that appears when you simply rerun the identical configuration. The point was to separate the effect of switching harnesses from noise produced by repeat runs. On a set of 45 hard tasks, the share of individual tasks that flipped between success and failure was a median 13 percent both when the harness was changed and when the same configuration was rerun. The researchers explained that small score differences alone are not enough to judge whether something has improved.
How large were the gaps in resolution rate
Across the remaining 447 tasks, excluding the 45 hard ones, Claude Code and mini-SWE-agent showed similar resolution rates. OpenCode trailed by as much as 9 percentage points. For one model, roughly half of that gap was concentrated in runs that hit a preset output limit and stopped partway through. What decided the outcome was not just the model’s capability but also how far the setup allowed the work to continue.
The number of tasks in the test set also mattered for reading the results. According to the researchers’ analysis, with 45 tasks the chance of detecting a true 13 percentage point difference in resolution rate was only about even. With 447 tasks, differences down to around 5 percentage points could be distinguished. In other words, it is hard to pick a coding tool based on a handful of success stories or a narrow score margin.
Why costs stretched to 3x
When comparing costs, the researchers used the same model, the same tasks, and the same price list. Even so, the cost per task differed by up to 3x depending on the harness. They pointed to two causes: the length of the system prompt and tool descriptions resent to the model at every step, and the number of steps taken to finish the job. Repeatedly sending long instructions, or walking through more steps, is what drives the cost up.
The findings also point to what users should weigh when putting a coding AI to work. Using the same model does not mean the cost is the same across tools. Resolution rate and cost per task have to be read together.
The public repository contains a data table logging 8,032 runs along with the analysis code. However, part of the full run logs and the experiment environment materials are missing, so reproducing every run exactly is difficult. Whether these gaps in resolution rate and cost hold up with other models, and in real work, is the next thing to check.