AI evaluation budgets must account for disagreement

An evaluation budget buys evidence for a decision. When the evidence depends on subjective human judgment, spending everything on more examples can leave disagreement about each example poorly understood. Technical leaders should choose the balance between breadth and repeated ratings according to the metric and the decision they need to make. A larger list of examples does not automatically answer the question that motivated the evaluation.

The Google Research discussion published on 31 March 2026 examines that trade-off. It explains that accuracy against a plurality label and metrics sensitive to the distribution of opinions can favor different allocations. The associated paper, Forest vs Tree: The (N, K) Trade-off in Reproducible ML Evaluation, has a publication date of 14 March 2026. The later author discussion is the March 31 event; it is not the paper's publication date.

Decide what the score must mean

Start by writing the decision the evaluation will inform. Comparing two candidate models is different from deciding whether a sensitive behavior needs human approval. A score can summarize evidence only under a defined interpretation. If the team has not agreed what an acceptable result means, choosing a rater count first gives the collection plan a precision that the decision itself lacks.

For subjective tasks, ask whether the product needs agreement with the most common judgment or a representation of differing judgments. Both can be legitimate targets. They answer different questions. A single aggregate label can be useful for the first while hiding information needed for the second. The score should carry the interpretation the team actually intends to use when deciding what to release.

That choice belongs to the people responsible for the product and evaluation. It should be settled before a collection vendor turns the request into a fixed number of ratings. The annotation plan can then explain why repeated judgments matter for this decision, or why broader coverage deserves more of the budget. An unexplained preference for either breadth or repetition leaves the buyer unable to assess what evidence is being purchased.

Preserve disagreement before aggregation

Repeated ratings can reveal that people interpret the same output differently. Keep the underlying judgments available when the intended metric needs that information. Aggregating immediately into one label can remove the variation the evaluation was meant to study. This is a recommendation about evidence design, not a claim that every product should optimize for a distribution of opinions.

Two distributions of judgments can produce the same winning category while reflecting different levels of agreement. The Google discussion uses this distinction to motivate attention to human variation. For a product team, the implication is to decide which differences could change the release decision. Collecting repeated ratings has a purpose when it helps answer that question, rather than simply making a benchmark appear more rigorous.

Disagreement also needs interpretation. Unclear instructions can create differences that do not represent the intended user population. A homogeneous group can agree while missing relevant perspectives. More ratings cannot automatically repair either problem. Specify who should judge the task and what the instructions mean before treating an observed distribution as representative of the people the product must serve. Those are prerequisites for useful collection, not results established for your application by the paper.

Allocate against the chosen metric

For planning, separate the number of distinct items from the number of judgments per item. Increasing one spends resources that cannot also be spent on the other under a fixed budget. The balance should follow what the metric needs to estimate. The research discussion supports that dependency; it does not supply a universal rule that ten raters, or any other fixed count, is enough for every subjective task.

A team can propose a limited initial collection to understand where uncertainty matters, then decide whether additional coverage or repeated ratings would better inform its chosen decision. That is an application of the research question, not a reported local evaluation. State which evidence would change the allocation. Otherwise, a preliminary choice can become a permanent purchasing rule without anyone checking whether it still serves the metric.

Keep release authority separate from statistical evidence. A repeatable model comparison does not decide the business's tolerance for harmful behavior. Someone still needs to own the acceptance rule and any human review requirement. The budget can improve the evidence available to that person; it cannot delegate the decision to a score whose consequences have not been agreed.

Executable answers change the problem

The strongest counterargument comes from tasks with objective, executable oracles. If correctness can be established by a reliable check against a defined specification, collecting many subjective opinions may add little to that correctness decision. Prefer the evidence that actually resolves the task. Human disagreement is not a reason to replace a suitable executable check with a panel of raters.

Mixed tasks require a finer boundary. A response can satisfy a formal condition while still needing judgment about tone or appropriateness. Evaluate the formal condition with its suitable check and define the separate human question. Do not assume that passing one establishes the other. Equally, do not apply the subjective-rating argument to every output merely because an AI system generated it.

AL+ offers custom software development for businesses defining application and evaluation responsibilities. Bring the release decision, the intended users and the behaviors that require judgment. A useful plan connects each collection expense to the uncertainty it can reduce. Choose breadth where coverage serves the metric, preserve repeated judgments where variation matters, and keep the person responsible for acceptance explicit. No annotation campaign or local model evaluation is claimed by this article.