Test the harm your AI feature can cause
An AI safety score cannot describe every application risk. Trace one illustrative abuse case through input, capability and impact to define useful checks.
By 
Paul. AI and agentic engineering
An AI feature should be assessed against the harm its complete application can cause. A model's safety evaluation is useful evidence about that model under the evaluation's conditions. It cannot establish that your retrieval, permissions, tools and approval screens prevent a harmful business outcome. Define the application threat model before treating a general score as release evidence.
This November 2026 retrospective returns to the week of 15 January 2025. Microsoft's authors submitted Lessons From Red Teaming 100 Generative AI Products on 13 January. Their report distinguishes product red teaming from safety benchmarking. The experience described is theirs, covering more than 100 products at Microsoft, not an AL+ testing record.
Name the system and the downstream impact
The paper's version-one full text sets out an ontology connecting the system, actor, tactics and techniques, reproducible procedures, weaknesses and impacts. It includes malicious actors and benign users who encounter failures unintentionally. The important shift is to connect a failure mechanism to an outcome in the surrounding system.
Its impact categories include security consequences and safety harms related to generated content. Those categories help avoid a narrow assumption that every AI problem is a clever jailbreak or a stolen credential. The product's purpose, audience and available capabilities determine which consequences deserve attention. A vocabulary organizes that investigation; it does not supply a risk estimate automatically.
For a business owner, this changes what to ask of a release assessment. “Did the model refuse an unsafe request?” and “Could this application disclose another customer's information?” are different questions. A satisfactory answer to one cannot silently stand in for the other. The second requires an account of data access and the application's actual actions.
One illustrative trace, from input to impact
Consider a fictional quotation assistant used by a supplier. It reads a customer's uploaded specification, retrieves internal product information and prepares a quotation for a staff member to approve. This example is explicitly illustrative. It describes neither a deployed AL+ integration nor a penetration test.
The actor is an external customer who can control the uploaded specification but has no legitimate access to another customer's negotiated terms. The input is a document containing instructions that try to redirect the assistant toward those terms. The relevant capability is retrieval from the supplier's information store. The candidate weakness is retrieval authority broader than the requesting customer's entitlement, combined with treating document content as instructions.
The possible impact is disclosure of another customer's terms in the proposed quotation. An approval screen may intervene before external delivery, but that does not establish that the confidential information never reached an unauthorized internal context or that the reviewer will notice its origin. The trace identifies what must be checked; it does not claim that every assistant with document input will behave this way.
A proposed assessment would preserve the actor's actual permissions, use controlled fictional records and observe whether the restricted information can reach the draft or an outward action. Success for the safeguard would mean that authorization prevents the retrieval or disclosure, rather than relying only on the model to reject persuasive text. These are proposed acceptance conditions, not new findings.
Choose mitigation at the boundary that failed
If retrieval returns records the actor should not be able to use, repair that authority boundary. A stronger instruction to the model may help behavior, but it does not explain why the data was accessible. If the draft is authorized internally but external delivery requires approval, make the approval meaningful: show what will be sent and which information the decision relies on.
The same trace can reveal several responsibilities without requiring a separate taxonomy for each screen. Access control governs available records. The assistant's handling of untrusted content governs how it interprets the document. Delivery controls govern what can leave the application. An assessment should retain the relationship between the observed weakness and the proposed remedy.
This is distinct from deciding whether a browser agent may download an executable, and from using AI to discover conventional software vulnerabilities. Those are worthwhile subjects, but they involve different capabilities and evidence. Here the question is whether a particular application path can carry untrusted input into an unauthorized information outcome.
Baseline model evaluations still matter
The strongest counterargument is scale. Product-specific investigation takes time, and baseline evaluations can compare model behavior systematically. They can help select candidates, detect changes and identify broad classes of undesirable output. Discarding them would remove useful evidence rather than improve the assessment.
Keep their scope visible. Record which model and configuration were evaluated, the tasks the evaluation covered and what result would change the product decision. Then add application-specific cases for capabilities and impacts that the baseline does not exercise. A strong baseline result can coexist with a dangerous permission mistake in the surrounding software.
The historical paper provides a framework and the authors' lessons, not a certification scheme or a complete list of future harms. Its versioned text anchors the January argument without treating later vendor documentation as contemporary evidence. A real release still needs investigation proportional to its actual data, users and consequences.
For AI agents, begin with one consequential path: who controls the input, what capability can act on it, which weakness could connect them, and who could be harmed. Turn that trace into an observable acceptance condition and retain unresolved limits. A safety score becomes more useful when it sits beside this application evidence, with each answering the question it can actually support.