An old checkout can still expose tomorrow's fix

An agent evaluation can show the correct old source files while still exposing information from the future. Checking out a historical revision changes the working tree; it does not establish that every readable reference, object or message belongs to that moment. A defensible coding evaluation needs an information boundary, not merely a base commit.

Looking back from November 2026, SWE-bench issue #465 is a concrete example. Jacob Kahn opened it on 3 September 2025. The community discussion that brought it wider attention on 11 September did not originate the report. The chronology matters because the project's response was already underway before that later discussion.

What the report actually establishes

Kahn presented public trajectory examples in which agents consulted repository history containing later fixes or descriptions of them. One concerned a pytest task; others concerned Django. The report is primary evidence of what the reporter observed and supplied, not an independent rerun in this article. No benchmark was executed and no private repository or hidden answer was accessed here.

A follow-up from a reporting collaborator separated trajectories judged to obtain the reference patch from those merely learning future information without a demonstrated link to it. The collaborator called the search preliminary, allowed for classification mistakes and said the team lacked an automatic detection method. Preserve those qualifications. Exposure to future information and proof that it changed a successful answer are different claims.

The confirmed documentary fact is that the public report contains specific examples and limitations. The broader allegation is that such leakage can distort scores. Those examples justify inspecting affected runs, but they do not establish the fraction of every leaderboard that was affected. A suspicious command, a contaminated environment and a result that would fail under isolation are three different findings.

A date belongs to the readable environment

Git can retain more information than the files currently checked out. References can name later commits. Reflogs can preserve movements and messages. An object store can contain content outside the intended historical view. This makes “offline” an incomplete description of isolation: the agent can encounter disallowed future information without making a network request.

The defensive requirement is to define which information belongs in the task. Historical context may be legitimate and useful for a regression investigation. Future fixes are not legitimate evidence when the test asks whether the system can derive that fix from an earlier state. The boundary must follow the evaluation's question, rather than an arbitrary ban on every use of version history.

Construct evaluation artifacts for that boundary and inspect what the agent can actually read. Treat repository metadata, mounted caches and preinstalled artifacts as inputs to review. Keep the authoritative answer and hidden tests outside the agent's accessible environment. These are design recommendations, not instructions for extracting a fix from somebody else's benchmark or a claim that a cleanup recipe was tested here.

The maintainer response changed the situation

A maintainer response on 4 September said the harness and images were being adapted. On 11 September, Carlos Jimenez reported that Verified images had been updated and the build process fixed through PR #471 in package version 4.1.0, preserving past history and tags predating the base commit. That response distinguishes a reported mitigation from the original exposure.

The same discussion records an initial maintainer search finding few matches for selected history commands and rejecting a wholesale rerun at that stage. Its method was narrow; it was not proof that other access patterns were absent. Conversely, the reporter's preliminary findings do not justify replacing that estimate with an invented global contamination rate. The disagreement is a reason to disclose method and scope.

The issue is now closed; the public API records closure on 24 March 2026. A February comment had said Verified and Multilingual were fixed while Multimodal work remained. Accessed on 9 November 2026, these records do not certify every cached image or every downstream evaluation. A closed issue is project status, not a receipt proving which artifact produced an earlier score.

Compare results with their environment attached

The practical purchase decision is whether a reported score answers your question about repository work. Ask for the model and agent configuration, environment version, image identity and evaluation date. If a run predates a mitigation or uses a retained artifact, its score needs that context. The right response may be a rerun or a qualified comparison, depending on the evidence available.

The strongest counterargument is that version history is part of real engineering. An agent that consults an earlier commit can be behaving usefully, and aggressively stripping history can make a task less realistic. That objection supports preserving permitted past context. It does not make a future solution legitimate in a test intended to measure independent repair from an earlier state.

A production assistant can also be useful even when a benchmark claim needs correction. Do not turn a defect in evaluation isolation into a claim that all model-assisted work lacks value. Equally, do not use a useful product experience to validate an unrelated leaderboard method. The two decisions need their own evidence.

For custom software development, a local acceptance task should state its allowed information and retain the environment identity with the result. Test the boundary as part of preparation, before comparing candidates. An old working tree supplies a date for the visible files. A trustworthy evaluation must account for the rest of what the agent could know.