Choose coding models on your own tasks, and keep the choice reversible

A coding model's position in a vendor's product range is a poor reason to build your workflow around it permanently. Claude 3.5 Sonnet's June 2024 introduction illustrated why: Anthropic presented a mid-tier offering that outperformed its previous flagship on its coding evaluations. The durable response is to keep model choice reversible and judge permitted candidates on tasks from your own repository.

That is an argument for a small, repeatable evaluation and a clear integration boundary. It is not a recommendation to change providers whenever a leaderboard moves. A switch only pays if the improvement survives your task requirements, review process and operating constraints.

Read the launch claim with its method

Anthropic's June announcement, whose page currently displays 21 June 2024, reported 64% of problems solved by Sonnet against 38% by Claude 3 Opus in an internal agentic coding evaluation. It also described lower cost and higher speed than Opus. These were vendor results about the models of that period, not today's ranking or an independently reproduced productivity gain.

The model-card addendum explains that the coding evaluation involved implementing changes to open-source repositories, with tests hidden from the model and an agentic loop in a sandbox without internet access. That method matters. A model answering a chat question without repository tools is not operating under the same conditions.

Both documents come from Anthropic. Two links do not make two independent measurements. They justify examining a candidate, not accepting its reported score as the expected success rate for your work.

Later history also needs its own date. Anthropic announced an upgraded Sonnet on 22 October 2024. Its reported results used other benchmark names and conditions. Do not place that upgrade in June or line up unlike percentages as if they formed one continuous measurement. From November 2026, the defensible conclusion is that a familiar model name can cover meaningful changes over time.

Build a small set that can change your decision

Start with about a dozen representative tasks, if that is enough to cover the work you actually assign. The number is a practical starting point, not a statistically justified sample size. Include an ordinary change, a bug requiring investigation and a task where touching unrelated code would be a failure.

For an illustrative ordering application, one task might ask for a validation change while preserving the existing error response. Another might require finding why a total is wrong without changing the public interface. These are proposed examples; no models were run on them for this article.

Write acceptance criteria before seeing outputs. Record required behaviour, forbidden shortcuts and the checks a reviewer will use. A green test suite is useful but cannot establish everything: an unnecessarily broad rewrite may pass while making maintenance harder. Include a human assessment of scope and clarity.

Protect the evaluation data. Use tasks and code you are permitted to send to each candidate. A model that cannot receive the relevant data under your requirements is not an available option, however strong its advertised score.

Compare the workflow, then isolate the model

Keep the repository revision, available tools and time budget comparable. Record the prompt, model identifier and settings that the service exposes. If one candidate receives search tools and another receives only pasted text, you are comparing two systems. That can be useful when choosing products, but label the result accordingly.

Count review effort and retries alongside accepted results. If a candidate generates a plausible patch quickly but repeatedly alters unrelated files, its response speed may contribute little to the completed task. Compare the total work required to reach an acceptable change rather than tokens alone.

Repeat borderline tasks when feasible and keep failures. A small set can guide a local decision without proving general superiority. Rotate part of it as the code and workload change, so optimizing prompts against remembered examples does not become the whole exercise. Preserve a stable portion to notice regressions.

Keep the integration boundary narrow

The tool that asks for a proposed patch should not need provider-specific details spread throughout the application. Put request construction, response parsing and error handling behind a small boundary. Keep credentials and data permissions attached to the selected provider configuration, rather than assuming every replacement has identical rights.

A common request shape does not guarantee equivalent behaviour. Tool calling, context limits and structured output handling can differ. Preserve those distinctions where they matter and validate them before switching. An adapter should make differences manageable, not conceal them until a task fails.

Keep fallback deliberate too. A temporary outage is not permission to send confidential code to another provider. If no approved alternative exists, pause that task and preserve its state. Availability and data authorization are separate decisions.

Portability also has a cost

The strongest objection is that a universal model layer can become a product of its own. Supporting hypothetical providers and every possible feature may consume more engineering time than a future switch would save. A team working well with one subscription need not buy several services merely to demonstrate optionality.

Use the subscriptions and permissions you already have. Isolate the few provider-specific operations you actually use, document their assumptions, and rerun the small task set when a material model or tool change is proposed. Keep the current choice when the evidence for switching is weak.

For an AI agent integration, the useful deliverable is a workflow whose quality can be checked and whose dependencies can be changed intentionally. A launch score can put a model on the shortlist. The decision belongs to the work the team can accept, review and maintain.