Most AI products ship with a latency chart, an error-rate alert, and a status page, and none of those will fire when the model quietly gets worse at the one task the product exists to do. The request still returns a 200.
The model under your product changes whether or not you ship
The clearest evidence that a hosted model is not a fixed dependency comes from a study published in the Harvard Data Science Review in March 2024. Lingjiao Chen, Matei Zaharia, and James Zou ran the same prompts against the March 2023 and June 2023 versions of GPT-4 and GPT-3.5. On a task as basic as classifying numbers as prime or composite, GPT-4’s accuracy fell from 84% in March to 51% in June. Both models made more formatting mistakes in code generation in June than in March. The authors’ conclusion is the one every founder should pin above the deploy button: the behavior of the same LLM service “can change substantially in a relatively short amount of time,” which they say highlights the need for “continuous monitoring of LLMs.”
That is the provider-side drift. The team-side churn is larger. In the startups we work with, the system prompt changes weekly, retrieval settings get swapped, temperature gets adjusted, tools get added, and the model version is upgraded when the vendor deprecates the old one. Each of those is a change to product behavior, and in our experience almost none of them pass through anything resembling a test. The engineering culture that would never merge a database migration without a test suite merges a prompt rewrite on the strength of three manual checks in a playground.
The reason is not negligence. It is that the instrumentation teams already have was built for a different failure mode. Uptime, latency, and error rate answer the question “did the system respond?” A language model fails by responding fluently and wrongly, and nothing in a conventional observability stack is pointed at that.
An uptime dashboard tells you the model answered. It cannot tell you the model answered worse than it did last week.
An uptime dashboard tells you the model answered. It cannot tell you the model answered worse than it did last week.
The frameworks already say test before and test regularly; almost nobody does
This is not a new idea that governance bodies have yet to reach. The NIST AI Risk Management Framework (AI RMF), published in January 2023, puts it in one sentence under its MEASURE function: AI systems should be “tested before their deployment and regularly while in operation.” The framework goes further in ways that read like a spec for a small harness. MEASURE 2.3 asks that performance be measured and “demonstrated for conditions similar to deployment setting(s)” — real traffic shapes, not synthetic demos. MEASURE 1.1 says to select metrics “starting with the most significant AI risks” and to document the risks that will not, or cannot, be measured. That last clause matters for a startup: a written list of what the harness does not cover is itself a governance artifact, and it costs an afternoon.
The security community says the same thing from its side. The OWASP LLM AI Cybersecurity and Governance Checklist, version 1.1 from April 2024, carries a section on testing, evaluation, verification, and validation whose first instruction is to “Establish continuous testing, evaluation, verification, and validation throughout the AI model lifecycle.” Elsewhere, under monitoring and response, the checklist also expects audit records to be secure and workflows, monitoring, and responses to be mapped. None of that requires a compliance function. It requires a test that runs.
The gap between the guidance and practice is documented. Stanford’s 2025 AI Index Report finds that “Evaluating AI systems with responsible AI criteria is still uncommon,” that the safety tests developers do run are typically internal and non-standardized, and that “Organizations acknowledge RAI risks, but mitigation efforts lag.” The same report, released on April 7, 2025, counts 233 reported AI-related incidents in 2024, a record and a 56.4% increase over 2023. Incidents are the lagging indicator; an untested change is the leading one.
For a startup, the reason to care is commercial before it is ethical. The enterprise buyer who eventually asks how you validate model changes is asking a question lifted directly from these documents. A team with a harness answers in a sentence and attaches the last run. A team without one begins the security review on the back foot.
The smallest harness worth building fits in a week
Our assessment is that a first evaluation harness is a one-week project for one engineer, provided the scope stays small. It has five parts, and the order matters.
- A golden set drawn from production. Take real prompts — anonymized where they must be — and pair each with an expected outcome: a correct answer, a required structure, a refusal, a citation that must appear. Synthetic prompts test what you imagined; production prompts test what the product does. Size the set in dozens, not thousands, and weight it toward the tasks that would hurt most if they regressed, which is MEASURE 1.1 applied to your own backlog.
- Graders that are deterministic wherever possible. Schema validity, required and forbidden strings, length, and refusal behavior can be checked with code. Reserve a model-graded rubric for judgment calls, and keep its prompt under version control like any other prompt.
- A trigger on every behavioral change. The harness runs in continuous integration on any change to a system prompt, model version, retrieval configuration, tool definition, or sampling parameter. A drop past a threshold blocks the merge. This is the step that turns an evaluation into a control.
- A scheduled run with no code change. Once a day, the same set runs against the production model. This is the only part of the stack that would have caught the drift Chen, Zaharia, and Zou measured, because nothing in your repository changed when the vendor’s model did.
- One line in the changelog. Every release records the harness score beside the version. Twelve months later that column is the evidence an acquirer’s diligence team, or your first regulated customer, will ask for, and it will already exist.
Two things stay out of scope on purpose. Do not try to measure everything; write down what you are not measuring, as the NIST framework asks, and move on. Do not wait for a benchmark to arrive from outside; the Stanford report’s finding that standardized evaluation remains uncommon means your own golden set is the standard for your product until someone offers a better one. The owner is whoever owns the prompt. The deadline is before the next feature ships, because every feature added without a harness enlarges the surface the harness will eventually have to cover.
Teams instrument for the failures they have seen and ship blind to the one they have not: a model that answers every request and gets quietly worse. The frameworks have told teams for years to test before deployment and regularly in operation, and the research shows the drift is real. A thin harness — a golden set, a grader, a trigger, a schedule, a changelog line — is the cheapest insurance on the roadmap, and the feature after it will be the first one you can prove did no harm.