Many MindedNew coding benchmark finds agents fully solve barely a quarter of real change requests
Researchers from Alibaba and several Asian universities have released Co-Design Bench, a test for coding agents built from real change requests taken from actual coding sessions. Agents must rework existing web apps, and the inputs can include images, audio recordings, videos or documents. Ten models were tested. GPT-6 Astra ranked first, followed by Claude Opus 5 and Qwen3.8-Max, with Grok 4.6 last.
The top score is the telling part. GPT-6 Astra met about 70 percent of a task's requirements on average, yet fully solved only 26.4 percent of tasks. It fully solved 41 percent of multimodal tasks and just under 18 percent of repository tasks. Every model got worse when a task needed three or four kinds of change at once, such as layout, design, interaction and content. Older benchmarks like WebCompass-edit show near-perfect scores, a gap of more than 35 percentage points. The researchers chose cases where users rephrased requests several times or rolled back changes, so the tasks skew toward the hard ones. METR separately found project leads' acceptance of AI-generated code ran roughly 24 points below SWE-bench Verified scores.
Why it matters: software is an input to nearly everything else we want to make cheap, and the promise is that agents will write it at near-zero cost. This result says the last stretch still needs a human, so the savings are real but smaller than headline benchmarks suggest. It is a friction signal about measurement as much as capability.
Caveats: each task is one fixed request, so follow-up questions and mid-task corrections are not tested. The AI judges, one testing the app in a browser and one re-checking positive verdicts, can make mistakes. The smaller Qwen3.8-27B placed second on larger overhauls, but the authors draw no general conclusion from that, and its score swung between 75 and 56 percent depending on the harness. Another recent study found little benefit from elaborate harnesses, so the environment effect is not settled.
Source: The Decoder