METR’s developer trial separated felt speed from completion time
METR’s early-2025 randomized trial found slower completion in familiar repositories. Its value is a measured workflow result, with explicit limits on scope.
By 
Paul. AI and agentic engineering
A developer can feel faster with an assistant while taking longer to complete the work. METR's randomized study made that disagreement measurable in one specific setting: experienced contributors working in repositories they already knew. The practical lesson is to compare completion effort under a clear acceptance standard before turning perceived assistance into a delivery forecast.
This November 2026 retrospective concerns the original early-2025 trial. METR's announcement is dated 10 July 2025; version one of the paper was submitted on 12 July. The announcement now links to a February 2026 update. That is later evidence, not part of the original result examined here.
Permission was assigned to tasks
The original trial involved 16 developers and 246 completed tasks, with 136 assigned to AI-allowed and 110 to AI-disallowed conditions. Participants had an average of five years' experience in their projects. They supplied real issues and forecast completion times before assignment. Randomization concerned permission to use assistance, rather than a requirement that every allowed task receive the same model or prompting strategy. These details come from the v1 methods and results.
That design is useful because the comparison does not simply ask enthusiastic adopters whether they are productive. Assigning conditions after task definition limits the opportunity to choose easy work for assistance and difficult work for the comparison. It also keeps the unit of analysis close to a decision a maintainer actually faces: finishing an issue in an existing project.
The treatment remains a workflow package. Tools, the developer's choices and the surrounding repository all contribute to the outcome. A result for permission to use tools cannot isolate the intrinsic quality of every model involved. Nor does it establish what a mandatory, carefully optimized procedure would have achieved with the same participants.
The historical tools were mainly Cursor Pro with Claude 3.5/3.7 Sonnet. Naming them identifies the study setting; it is not a recommendation or an execution of those tools here. Replacing that historical setting with an unspecified November 2026 assistant would change the question. A current adoption decision needs evidence for the current workflow.
A point estimate needs its uncertainty and its denominator
METR reported 19% longer implementation time with AI allowed. Participants had predicted 24% shorter completion time and afterward estimated a 20% reduction. Those are three different quantities: a measured treatment estimate, a forecast and a retrospective belief. They should not be combined into a single productivity score or treated as three independent measurements of actual speed.
The paper's Appendix D describes regression on log completion time, adjusting for forecast difficulty, and transformation of the coefficient and its 95% confidence interval back into a time ratio. It also reports a ratio-of-means robustness approach using bootstrap intervals. The 19% headline is therefore an estimate under a stated analysis, not a claim that every contributor slowed by exactly that amount.
Statistical uncertainty and applicability are separate limits. An interval around an estimated effect does not make a selected group representative of all software work. Even a precisely estimated effect in familiar repositories would leave open the result for a beginner, an unfamiliar codebase or another kind of task. Conversely, a small sample does not justify discarding random assignment and replacing the outcome with preference surveys.
The outcome also deserves accurate language. Developers reported implementation time and recorded their screens; the research is not an omniscient measurement of every cost a company incurs. A business considering adoption may care about reviewer effort, elapsed delivery time or cognitive strain as well. Those are additional outcomes to define, rather than silently attribute to the reported percentage.
Familiarity changes the comparison
For a maintainer, knowing where a change belongs can be a substantial advantage. An assistant's plausible suggestion may still require explanation, checking and adaptation to project conventions. This is an interpretation of why the workflow matters, not a new causal estimate of how much each activity contributed to the result.
It follows that code volume is a weak substitute for accepted work. Generating a large patch could help, or could increase the material a contributor has to inspect. A proposed local comparison should retain the acceptance standard and count effort spent resolving unsuitable suggestions. Otherwise the evaluation can reward producing material that another person later has to remove.
Keep abandoned attempts visible too. If a team measures only accepted assisted patches, it can lose the time spent on assistance that was ultimately discarded. The comparison should describe when work starts, when it is considered complete and how interruptions are handled. These are recommendations for a future evaluation; no AL+ experiment or benchmark was run for this article.
Better tools are a serious counterargument
The strongest objection is that early-2025 behavior need not persist. Different tools, better context or a more effective way of using the same tools could change the result. METR explicitly cautioned against generalizing to most developers or future systems. That is a reason to repeat a relevant comparison, not to reinterpret the original estimate as a finding about today's models.
Assistance may also be worthwhile for an outcome other than speed. A developer could value reduced strain or help exploring an unfamiliar approach while accepting extra completion time. Make that trade explicit. A benefit in comfort does not prove shorter delivery, just as a time penalty does not prove that every use should be banned.
This trial asks a different question from METR's earlier task-horizon metrics: what happened to completion time when familiar human contributors were allowed assistance? For custom software development, start with that concrete decision boundary. Retain assistance where the accepted result and its review cost justify it, and revise delivery expectations when measured effort disagrees with the feeling of progress.