Top Coding AI Completed Just 26.4% of Real-World Edit Tasks

Researchers from Fudan University, Alibaba Group and other institutions released a paper on October 26 introducing Co-Design Bench, a benchmark that tests how well coding AI can modify existing web apps to meet user requests. In an evaluation of 10 models under the same conditions, even the top performer, GPT-6 Astra, fully completed just 26.4% of tasks.

How many real requests did AI finish?

The benchmark draws on requests that AI struggled with in actual use. The researchers selected 273 tasks from requests across 26,314 coding sessions. These included 100 targeted edits that could involve images, audio, video or documents, and 173 project-level edits requiring changes to multiple components at once.

The evaluation measured both whether models met individual requirements and whether they completed each task in full. It checked 2,331 expert-defined criteria against the code and the running app’s interface, then used a separate AI verifier to recheck judgments of full or partial compliance. GPT-6 Astra met an average of 69.9% of the criteria per task, but satisfied every criterion on just 26.4% of tasks.

That means roughly 3 out of 4 tasks still had unmet requirements. When users requested several changes at once, there was a wide gap between handling some requirements and finishing the entire request.

Why did an earlier benchmark yield 99.4%?

Some models scored much higher on an existing benchmark. In a comparison test, the researchers had AI agents, which carry out work autonomously, tackle 150 WebCompass-edit tasks. The average feature-level scores were 95.1% and 99.4%.

The two benchmarks differ in their tasks and scoring methods. The high scores on the existing benchmark reflect assessments of individual features, while Co-Design Bench covers complex editing requests drawn from real user sessions. It also reports the rate of meeting individual criteria separately from the rate of completing entire requests. For anyone considering delegating work to coding AI, both measures matter: whether individual features work properly and whether all requested changes get done.

Can you run it yourself?

The evaluation code and examples are public, but the released task set is smaller than the 273 tasks used in the paper. It includes only targeted edits, leaving out project-level tasks that require changes to multiple components at once. Running the evaluation also requires additional tools and an appropriate environment.

The central takeaway is that choosing a coding AI means looking beyond its score to whether it finishes the entire request. For users considering it for work, a key comparison is whether the benchmark resembles the edits they regularly need. Questions to watch include how closely results on the public subset match the full evaluation, and whether requests in Korean produce similar completion rates.