The first serious “why did it do that?” arrives after the fact: a customer with a screenshot, a regulator with a letter, an acquirer with a diligence list. Whether you can answer depends entirely on what your system wrote down before anyone thought to ask.
The question arrives with a bill attached, and the record decides who pays it
In Moffatt v. Air Canada, decided by British Columbia’s Civil Resolution Tribunal on February 14, 2024, a grieving customer asked the airline’s website chatbot about bereavement fares. The chatbot told him he could apply for the reduced fare within 90 days of ticket issuance, after travelling. The page it linked to said the opposite. He booked two flights for a combined $1,630.36, applied within the window, and was refused. The airline later acknowledged the chatbot had given misleading information, then argued in its defence that it was not responsible for what the chatbot said. The tribunal called that “a remarkable submission,” observed that the airline was in effect suggesting “the chatbot is a separate legal entity that is responsible for its own actions,” and found that Air Canada “did not take reasonable care to ensure its chatbot was accurate.” The total award was $812.02.
The number is small. The mechanism is not. The case turned on two things a startup can control: what the system actually said, and whether the company could show it had taken reasonable care. Air Canada had no answer to either. A product that records each conversation with the model version, prompt version and retrieved policy text that produced it can answer the first question in minutes and the second with a review log. A product that does not is left arguing about a screenshot the customer took.
Regulators are writing the log into law, and your customers will pass the obligation to you
The European Union has made the record a legal feature of the product. Article 12 of the EU AI Act requires that high-risk AI systems “shall technically allow for the automatic recording of events (logs) over the lifetime of the system,” with traceability “appropriate to the intended purpose” and events relevant to identifying risk, supporting post-market monitoring and monitoring operation.
Annex III is the Act’s list of high-risk uses, such as biometrics, hiring and credit. For the remote biometric identification systems listed in Annex III, point 1(a), the article spells out a floor: the period of each use with start and end time, the reference database checked, the input data that produced a match, and the identity of the natural persons who verified the result.
Article 26(6) then requires deployers — the Act’s term for the business that runs the system, as opposed to the vendor that builds it — to keep those automatically generated logs “for a period appropriate to the intended purpose of the high-risk AI system, of at least six months.”
Most startups reading this are not building Annex III systems. The point is the shape of the obligation. The deployer — your enterprise customer — must keep logs it may not generate, which means the duty flows back up the contract to the vendor that does. A B2B AI product that cannot hand its customer a per-event record with timestamps, inputs, model identity and human review is a product that enterprise procurement will eventually decline.
The United States is reaching the same place through older law. The Consumer Financial Protection Bureau’s (CFPB) Circular 2023-03, issued September 19, 2023, confirms that a creditor using a complex algorithm may not fall back on the sample checklist of reasons for a credit denial in Regulation B to satisfy its obligations under the Equal Credit Opportunity Act (ECOA) “if those reasons do not specifically and accurately indicate the principal reason(s) for the adverse action.” The circular is blunt about the gap it is closing: the duty applies to “technology that creditors may not understand sufficiently to meet their legal obligations.” If a model in your product influences a credit decision about a person, the specific reason for that decision has to be recoverable afterwards, which is a provenance requirement wearing a consumer-protection label.
The question “why did it do that?” is only answerable with evidence collected before anyone thought to ask it.
The question “why did it do that?” is only answerable with evidence collected before anyone thought to ask it.
Enough provenance is sixteen fields, and the standard already names them
The reason day-one provenance is nearly free is that the schema has been written for you. The OpenTelemetry project’s semantic conventions for generative AI spans, published in version 1.27.0 in August 2024, define one span, a timed record of a single operation, per model call. Three attributes are required: the operation name (gen_ai.operation.name), the system (gen_ai.system) and the requested model (gen_ai.request.model). The recommended set adds the model the provider actually responded with (gen_ai.response.model), the response identifier, temperature and top-p (the sampling settings), maximum tokens, input and output token counts and the finish reasons (why the model stopped generating). The gap between the requested model and the responded model is the most useful field for the failure that surprises teams most often: it is how you learn, months later, that the vendor served a different model than the one you tested. The conventions were marked experimental at that version, and two of the conventions’ authors wrote on the OpenTelemetry blog in December 2024 that the goal is to make it easier to monitor, troubleshoot and debug AI applications, with prompt and completion content captured as events only when the operator turns it on.
That opt-in is deliberate. The specification notes that “end users of GenAI applications may input sensitive information or personally identifiable information (PII)” and requires that instrumentations “MUST offer the ability to turn off capture of prompts and completions.” In our experience the right default for a regulated customer base is to capture the content, encrypt it at rest, scope access, and set retention to the longest obligation you can foresee, with six months as the floor the EU has already set. The provenance record we ask client teams to write on the first day a model call ships is sixteen fields in four groups:
- Identity of the call: request identifier, timestamp, user or tenant, application version and prompt template version.
- Identity of the model: provider, requested model, responded model and response identifier, exactly as the conventions name them.
- Inputs the model saw: the rendered prompt, the identifiers and versions of any retrieved documents, and the tool calls made with their results.
- What came back and what happened next: the output, the finish reason, the parameters, and the name of any human who reviewed or overrode it.
Written as structured spans against an existing tracing library, this is a few hours of engineering and a storage line item measured in cents. Retrofitted after an incident, it is a different project: the vendor has rotated the model, the prompt template lived in code with no version tag, the retrieved documents have been re-indexed, and the engineer who wired the pipeline has left. You are no longer reading a record; you are reconstructing one, under deadline, for a reader who is entitled to doubt it.
Provenance is not an observability nicety for later; it is the evidence that lets a small company say “here is what the system did, and here is why” on the day someone with authority asks. The fields fit on an index card and the standard already exists. Write them down on day one, keep them for six months at least, and the incident that would have been a reconstruction becomes a lookup.