A feature benchmark needs a versioned grading contract

A coding-agent result depends on the rules that turn a submitted change into a pass or failure. For feature tickets, those rules need to account for legitimate changes to existing tests. Version the grading contract alongside the task corpus. Otherwise, the same implementation can acquire a different result while the headline still appears to describe the same experiment.

The Agents on Rails Stage 2 article is dated September 9, 2026. Its current text contains an explicitly dated September 15 update changing how test directories are treated during feature grading. The September 21 follow-up explains regrading and a separate isolation correction. These later accounts are hindsight, and their revised results should not be presented as original September 9 measurements.

Feature work can change what a test expects

The Stage 2 account describes feature tickets against Fizzy. The grading discussion contrasts restoring test directories with allowing feature-related changes to remain. That distinction matters because an existing assertion may describe behavior a requested feature is supposed to change. Restoring the old assertion can make the grader reject the intended behavior.

The engineering question is what the ticket author intends to keep invariant. Some existing behavior should survive a feature; some must change for the feature to exist. A fair contract needs a way to distinguish the two. “All tests stay green” is incomplete if it leaves unanswered which version of the tests defines completion.

Allowing edits is not equivalent to accepting every edited test. A candidate could weaken an assertion without implementing the requirement. The evaluator still needs independent evidence of the requested behavior. The right boundary protects the meaning of the task while allowing the normal work required to fulfill it.

This is why the task description and hidden checks need to agree. If the description permits a behavioral change but the check enforces the old behavior, the result measures a conflict in the evaluation. If the description is too vague to settle that question, the evaluation owner has a specification problem to resolve before treating failure as a capability finding.

Sealed tests have a legitimate purpose

The strongest counterargument comes from atomic tasks. When a fixed test suite defines the requested behavior, restoring it protects against a candidate that changes the definition of success. A sealed suite can be a sensible anti-gaming boundary. A feature evaluation should explain why its boundary differs instead of treating protection of the grader as inherently misguided.

The distinction follows the job. An atomic task can ask for a specific implementation under unchanged expectations. A feature ticket can require new behavior and corresponding revisions to old expectations. Applying one policy to both can confuse consistency with fairness. A consistent rule is useful only when its meaning fits the task being measured.

For reviewers, this suggests a concrete check of the contract. Identify which evidence the candidate may change and which evidence remains independent. Explain why the separation preserves the intended requirement. That explanation makes the policy reviewable without requiring the reader to infer its purpose from a score table.

Regrading changes the interpretation of an old run

The September 21 explanation says recorded Stage 2 runs were replayed under the revised feature-grading rule. A replay under a different rule is a new judgment of retained work. It should be distinguishable from a fresh generation attempt and from the original judgment.

A useful results record would therefore retain the task version, submission identity and grading-policy version together. The dates of generation and judgment can differ. Readers need enough information to understand whether a change in outcome came from different work or a different interpretation of the same work. A percentage without that relationship loses part of its meaning.

Corrections can improve an evaluation. Preserving the earlier contract is still useful because it explains what changed and lets readers assess the correction. The aim should be traceability rather than the appearance that the corrected rule had always been in force. A publication date at the top of a mutable page cannot provide that history on its own.

Isolation belongs beside the grader

The follow-up also reports an isolation problem and subsequent corrected runs. That is the authors' account of their benchmark environment, rather than a local experiment reproduced here. It supplies another reason to attach execution conditions to results: the meaning of a task depends on the resources a candidate could actually use.

For an evaluation owner, permitted resources are part of the specification. If an environment makes additional information available, the result may answer a different question from the one advertised. Changing the environment can be necessary, but the resulting run needs its own conditions. Combining unlike runs without explanation would hide the distinction readers need to interpret them.

That observation does not justify a purchasing recommendation or an overall ranking of coding models. These are task-specific runs under the authors' stated setup. A maintenance team choosing its own process would need evidence matched to its work and constraints. A vendor-scoped benchmark can inform questions without answering every operational decision.

For software maintenance, keep the same discipline when accepting agent-written features: define which behavior may change and which independent evidence establishes completion. Version the task and its grading contract together. A result becomes more useful when the reader can tell exactly what earned it.