Define better before asking a model to optimize code

A request to make code faster needs a fixed answer to two questions: what behavior must remain correct, and what runtime work is being measured? Without those boundaries, a model can produce a sophisticated-looking implementation whose speed comes from changing the task. Accept optimization only after the candidate preserves the required behavior and improves the specified workload.

Max Woolf's experiment, published on 2 January 2025, explores repeated requests for better code. Discussion on 3 January drew attention to the results. Looking back from November 2026, the important distinction is between a proposed optimization and evidence that it works. Woolf's measurements remain his reported experiments; none were reproduced by AL+ for this article.

Preserve the meaning before comparing the time

Woolf's task involves finding the difference between extrema among numbers whose decimal digits satisfy a sum condition. He reports a generated revision that substitutes bit operations and computes an incorrect digit sum. That is a semantic change. Its runtime cannot establish an improvement to the original operation, regardless of how advanced the implementation appears.

The broader mechanism is easy to miss during review. Shifting by four bits walks through base-sixteen groups. Extracting decimal digits involves a different representation. Both operations can be described with words such as digits or chunks, but those labels do not make their outputs equivalent. Read the actual transformation before accepting its performance explanation.

For an optimization proposal, retain an independent correctness oracle: a trusted way to determine the required answer. It might be a simple reference implementation together with explicit rules for edge cases. The oracle should remain outside the candidate's edits. Its authority comes from matching the specification, not from merely being older than the optimized code.

Specify the runtime job

Imagine a hypothetical function that scans a collection and returns qualifying minimum and maximum values. The measurement contract should say whether input construction is included, whether the function runs once or repeatedly and how empty results are represented. The candidate must not gain an advantage by silently receiving less work or returning less information.

Preserve representative input sets when comparing revisions. Generating different random values for each candidate can change the amount of work and the expected answer. Fixing inputs does not mean designing them to favor one algorithm. Include the sizes and distributions that reflect the intended use, along with meaningful boundaries.

State the runtime environment too. Interpreter version, available cores and relevant dependency versions can affect a result. A measurement on one machine supports a conclusion about that configuration and workload. It does not establish the same speedup in a service with different memory limits or an entirely different request pattern.

Separate algorithm cost from moved work

An algorithmic change can reduce the number of operations needed. Precomputing a table can instead move work to setup. Both may be useful, but they answer different cost questions. A table reused across many calls may be sensible; rebuilding it for each request could reverse the apparent benefit.

Count the lifecycle that the application actually pays for. A long-lived worker and a short command have different opportunities to amortize initialization. If an optimization requires retaining a large table, include memory and invalidation in the decision. A shorter timed function body is insufficient evidence when its preparation has disappeared from the measurement.

Parallel execution needs the same accounting. Starting workers, transferring data and combining answers are work. More processes can help a substantial task while making a small one slower. Inspect whether the chosen workload has enough computation to justify those costs before treating parallel syntax as a performance argument.

Warmup changes the question being answered

Just-in-time compilation introduces another boundary. A first call may include compilation while subsequent calls use already compiled code. A warm measurement and a cold measurement can both be useful, provided the report says which one it answers. Mixing them across candidates makes the comparison misleading.

Woolf's account discusses JIT precompilation and process overhead. Those details belong to his experimental setup, not a universal ranking of optimization techniques. For a proposed application change, define whether startup, first request or steady-state throughput is the decision being made. If more than one matters, report them separately rather than selecting the most flattering number.

Keep repeated measurements comparable and inspect variation. A tiny apparent difference may not justify a new dependency or a less understandable implementation. The commercial question is whether the improvement matters under the service's real operating conditions, including the cost of maintaining and diagnosing the change.

Exploration can begin before a full harness exists

The strongest counterargument is that asking a model for ideas can be useful before building a benchmark. A suggestion to avoid repeated conversion or move invariant work may identify a promising direction quickly. Requiring a large evaluation project before any exploration would waste time on small changes.

The distinction is between generating a hypothesis and accepting a speed claim. A developer can inspect suggestions, discard semantic mistakes and choose one candidate worth measuring. A simple task may need only a small comparison and a direct correctness check. The evidence should fit the claim and consequence, rather than imitate an elaborate benchmark suite.

This question also differs from whether an agent can satisfy fixed supplied tests. Here, the focus is workload invariance and the meaning of faster. Passing correctness checks is necessary for an optimization claim, but timing a different job still cannot establish that claim.

For custom software development, retain the reference behavior, define the paid runtime work and compare the exact candidate you intend to release. Ask a model for alternatives as freely as useful. Call one an optimization only when the unchanged task, measured under stated conditions, supports that conclusion.