Passing the supplied tests is a bounded objective

A coding loop told to make fixed tests pass searches for an implementation that satisfies those tests. Its success says something precise about that objective. It does not establish that the tests express the intended behavior or that the resulting implementation is acceptable outside the examples supplied.

David Winterbottom's 6 October 2024 proof of concept makes that boundary unusually clear. It feeds tests to an LLM, runs the generated application code and uses failures to request another version, with a bounded number of attempts. The author also reports a deliberately contradictory example that reaches an absurd passing solution. Simon Willison discussed the demonstration on 13 October, during the historical week examined here. This November 2026 retrospective examines the mechanism, not general developer productivity. No hands-on reproduction was conducted for this article.

Decide what the loop is allowed to optimize

The relevant specification has at least two parts: observable behavior and constraints on implementation. Tests can express examples of the first. They may leave the second largely implicit. A function that recognizes a few test inputs can satisfy those examples while failing to implement the rule a human reader inferred from the task.

This possibility does not require malicious intent or a theory about the model's motives. The search receives a stopping condition and feedback. If the stopping condition is passing tests, a candidate that exploits an omission can meet it. Calling the result clever or dishonest adds little to the engineering diagnosis: the acceptance condition did not rule it out.

Before running such a loop, write down the behavior that should generalize beyond the visible examples. Include invalid inputs where they matter and any restrictions on state, external access or special cases. If the requirements conflict, resolve the conflict rather than asking the loop to find a surprising way to satisfy it.

Winterbottom's example is useful precisely because it is deliberately extreme. It illustrates a mismatch between the supplied checks and an ordinary understanding of the intended operation. It is not a representative failure-rate measurement for coding assistants, nor evidence that all generated code will behave that way.

Protect the measuring instrument

A fixed test set should remain fixed during the attempt being evaluated. If the generator can weaken an assertion, change the runner or replace expected output, the meaning of “passed” has changed. Protecting the test files is necessary, but inspect other ways the generated code could influence the test environment as well.

Keep the candidate's permitted edits narrow enough to review. Changes to dependency loading, environment variables or shared setup can alter how tests behave without touching the test file itself. Run candidate code with only the permissions required for the task. A passing test is not an authorization to use credentials or reach unrelated services.

Record which candidate was tested and which checks ran. A report from an earlier version cannot establish acceptance of a later edit. If an attempt fails to finish or a tool fails before tests run, classify that as incomplete evaluation. Treating missing output as a pass would defeat the purpose of having an objective check.

Bound attempts and resource use. Repeatedly generating code may consume more time without clarifying a contradictory requirement. A useful stopping rule includes a route back to the person responsible for the specification. The failed attempt can then expose a missing rule instead of merely prompting another guess.

Reserve evidence the loop has not seen

Unseen cases can help distinguish learning the intended rule from matching the supplied examples. Build those cases from the same specification before inspecting candidate behavior. Include boundaries that matter to the task, rather than unrelated puzzles added after a disliked answer appears.

A holdout is only unseen while the optimization process cannot use it. Once its failures are fed back into further generation, it has become part of the visible objective. That can improve the implementation, but subsequent acceptance needs to acknowledge the changed evidence. Reusing the same cases indefinitely does not preserve an independent check.

Holdouts also inherit specification limits. A second set of mistaken expected answers supplies more checks of the wrong rule. Human review should examine why the expected behavior is correct, as well as whether the code produces it. Where a useful invariant exists, checking that relationship across inputs can complement examples without claiming exhaustive proof.

For an illustrative data conversion, acceptance might require preserving designated fields and rejecting malformed records. A visible example can show the required shape; independent cases can exercise omitted fields and boundaries. This is a proposed evaluation scenario, not code written or tested by AL+. Its purpose is to show what must be specified before the loop runs.

Inspection remains part of acceptance

The strongest counterargument is that small, tightly specified transformations may be served well by a fixed-test loop. If the domain is narrow, the expected behavior clear and the implementation easy to inspect, the loop can be a reasonable search mechanism. Requiring a large evaluation framework for every small task would waste effort.

The appropriate response is proportional acceptance. Inspect whether the implementation expresses the rule directly or merely encodes the examples. Check the restrictions that tests cannot cheaply establish. For a tiny pure function, that may be a short code reading with a few meaningful cases. A change touching durable data or permissions needs evidence appropriate to those consequences.

Keep implementation quality concrete. Ask whether the next maintainer can understand and change the rule, whether hidden state affects results and whether dependencies are justified. “Clean code” alone is too vague an instruction to repair an objective that permits the wrong behavior.

A model-comparison exercise asks which model handles selected repository tasks better. This question is narrower: what exactly does a particular loop's success certify? For a custom software change, define that acceptance boundary before running the loop. A passing result can be valuable evidence when its scope is clear; the business still needs a justified decision that the implementation does the intended job.