Evidence-Based AI: When the Answer Is Not Enough

2025-09-24

Executive Summary

• In environments with compliance requirements — ISO, GMP, FSSC, HACCP — AI doesn’t only need to be correct. It needs to be explainable based on specific evidence, and that evidence must be traceable and verifiable.
• “The AI doesn’t know why” is not an acceptable answer in an audit context. The organization — not the AI — bears legal responsibility for decisions and actions, whether AI participated or not.
• Building AI with evidence capability is not only a technical requirement — it is a risk governance requirement, and it demands a very different foundation than a standard chatbot.
• Organizations evaluating AI for operational processes need to ask: can the answers AI provides be verified — not just whether they are accurate?

The question no one wants to hear from an auditor

A situation familiar to any Quality Director in a manufacturing business going through an audit.

The auditor asks: “On what basis was the decision made to release this batch for shipment?”

The correct answer needs to include: the specific inspection results, the applicable standard, the person who confirmed, the time of approval, and the supporting documentation.

Now add one element to the scenario: an AI system participated in the analysis of the inspection results before the responsible person signed off.

The auditor asks further: “What data did AI use for this analysis? Which version of the standard was in effect at the time? Can the AI’s analysis output be reviewed?”

If the answer is “the system doesn’t retain that” or “we’re not sure what data the AI used” — that is a serious compliance gap. Not because AI was wrong, but because AI was used in a process with traceability requirements and left no trace.

Why unevidenced AI is a risk in regulated environments

In standard manufacturing operations, a confident but incorrect answer can cause problems. In environments with ISO, GMP, or food safety standards, an answer that cannot be verified — even if correct — is also a problem.

The reason comes from the nature of compliance itself.

Compliance doesn’t only require doing things correctly — it requires demonstrating that they were done correctly.

This is the core distinction. An organization can have excellent processes but still fail an audit if the records demonstrating those processes were followed correctly don’t exist.

When AI participates in any step of a process with compliance requirements — from analyzing data, to suggesting decisions, to generating documentation — AI becomes part of the audit trail. If AI cannot create an audit trail, it is creating a gap in the organization’s compliance record.

More specifically, the concrete risks:

Risk 1 — The decision cannot be reconstructed. If a problem occurs with a product batch, an investigation needs to reconstruct the chain of decisions that led to that batch being approved. If AI participated in the analysis but didn’t retain its input data and analysis output, the chain cannot be completely reconstructed.

Risk 2 — Data version at time of decision is unknown. Which data did AI analyze, at what point in time, against which version of the applicable standard? If this isn’t recorded, and the standard has since been updated, it becomes impossible to demonstrate that the decision at that time was correct under the standard then in effect.

Risk 3 — Unclear accountability. AI is not a legal entity. The organization is responsible for every decision made, whether AI participated or not. If there is no clear record of AI’s role and the human review step that followed, the boundary of responsibility becomes ambiguous — which is precisely where auditors focus most of their questions.

What compliance environments require from AI

For AI to operate within a compliance-regulated environment, certain specific requirements need to be met — requirements that go beyond “producing correct answers.”

Input traceability

AI must be capable of recording and retaining: what question or task was posed, which data was used as input (including version and timestamp), and which standard or rule set was applied.

This isn’t only a technical log. It is evidence of the conditions under which a decision was made.

Output explainability

AI doesn’t only provide a result — it needs to be capable of explaining why that result was reached, based on which specific evidence, at a level that allows a person with authority to evaluate the reasoning.

This doesn’t mean AI must explain itself in technical AI terms. It means the answer must come with specific grounds: “inspection result X showed Y, cross-referenced against standard Z version V, leading to this assessment.”

Clear human review checkpoint

In every process with a compliance requirement, there must be a clearly defined checkpoint where a person with appropriate authority reviews and confirms — or rejects — the AI’s output before it is recorded as a decision.

AI does not sign off. The person with authority reviews the AI’s analysis and makes their own decision. That must be clearly recorded in the documentation.

Complete audit trail

The entire chain — from input data, through AI analysis, to human review, to final decision — must be traceable, stored, and retrievable on demand.

The AI’s audit trail must be part of the overall process audit trail, not a separate technical log that exists in isolation and that no one outside IT knows how to read.

Evidence, trace, and audit trail — three distinct concepts

These three terms are often used interchangeably, but they have specific meanings in compliance contexts.

Evidence is specific proof that an event or result occurred. Examples: a temperature measurement result, an incoming inspection form, a photograph of a visual inspection result. Evidence is what an auditor requests to see.

Trace is the ability to follow a chain of events backwards: from the release decision on this batch, backwards to the inspection results, to the incoming raw material data, to the person who performed each step and the person who approved it. Trace is the basis of traceability.

Audit trail is a systematic record of who did what, when, with what result, and if something changed, who changed it and when. An audit trail is what makes it possible to conduct an audit without interviewing every person involved.

When AI participates in a process, all three must encompass AI’s activity: what evidence AI received as input, which chain it traced through, and whether the audit trail captures enough to reconstruct the decision sequence.

An appropriate approach for manufacturing with compliance requirements

Not every AI application in a business requires the same level of evidence. A chatbot that helps employees locate a procedure faster doesn’t need a complex audit trail. But AI that participates in decision analysis, suggests actions, or generates documents with compliance significance is a different matter entirely.

A practical principle for prioritizing: the level of evidence AI needs to produce is proportional to the consequence if AI is wrong and no one catches it in time.

Consequence of AI misidentifying the procedure a user should look up: the employee spends a few extra minutes finding the right one. Low.

Consequence of AI incorrectly analyzing quality inspection results, leading to a non-conforming batch being released: product recall, compliance violation, legal exposure. Very high.

Evidence requirements need to correspond to this consequence level.

This also means organizations don’t necessarily need to apply the most rigorous evidence requirements to every AI application. But for AI in high-consequence processes — particularly in quality control, approvals, and traceability — evidence is not optional.

Conclusion — The right question before deploying AI in a compliance environment

Before AI is introduced into any process with compliance requirements, one question needs a clear answer:

“If an auditor asks about this decision, can we explain what AI analyzed, based on what data, under which version of the applicable standard, and who reviewed and confirmed before the decision was recorded?”

If the answer is uncertain — then regardless of how accurate AI may be, it is not ready for that compliance environment.

This is not a reason to avoid AI in manufacturing and operations. It is the requirement for deploying AI correctly — with evidence, with trace, and with clear accountability.

What is your organization preparing to make AI compliance-ready?

→ Take the AI Readiness Assessment — includes evaluation of governance and compliance readiness.

Further reading:

• AI Agents for Business: What They Are, What They Can Do, and What They Need (previous)
• AI Readiness: Why AI Alone Won’t Make Your Organization More Intelligent (pillar)
• AI Readiness: 6 Conditions That Determine Whether AI Delivers Operational Value (related)

This article reflects OKELAS’s understanding of compliance requirements in manufacturing and regulated industries. Specific requirements for audit trails and evidence vary by standard (ISO 9001, ISO 22000, GMP, FSSC 22000, and others) and by certification body. Organizations should consult qualified compliance advisors when applying AI to processes with certification requirements.