LLMs violate the determinism assumption behind unit testing by design, so exact-output tests are a poor fit for the model layer. Engineers often discover this when a test asserting one exact string passes once, then fails when the same answer is phrased differently. Fuzzing can create another fragile test, and teams may end up testing only the surrounding code.
The problem is the mental model, not the engineer. Unit tests assume identical input produces identical output every time. LLMs do not follow that assumption, so testing must take a different approach.
The Core Difference: Distributions, Not Values
A unit test checks for a particular return value. An LLM test instead asks whether the output distribution has desired properties: factual accuracy above a threshold, suitable tone, correct format, and no hallucinations.
This is a statistical rather than point-based claim. The assertion becomes "90% of outputs on this input distribution pass the quality rubric," not "this output equals X."
Meaningful LLM testing therefore needs multiple samples, a population of inputs, or both. One input and one run provide little signal. A pass rate across 100 inputs from the real distribution gives a more useful quality estimate.
What Unit Tests Can Still Catch
Unit tests still matter in AI systems, but they belong around the LLM: test the deterministic code, not the model itself.
Write unit tests for prompt construction, checking that a template creates the correct string for a given context; output parsing, checking well-formed and malformed LLM results; retrieval, checking expected document IDs against a fixed test index; and tool routing, checking that a mocked input reaches the correct tool.
These tests cover deterministic scaffolding. They run quickly, cost little to maintain, and catch bugs such as corrupted prompt context, parser edge cases, and retrieval changes that alter document ranking.
They cannot determine whether an LLM produces quality output from correct inputs. That requires evaluation.
Evaluation as the LLM Testing Layer
Evaluation fills the role unit tests serve for deterministic code, adapted to non-deterministic behavior. It checks output quality rather than exact output, using a rubric or reference, a scoring function, and enough samples for a stable estimate.
A practical LLM evaluation harness has three parts:
A test dataset: representative inputs with reference outputs or rubric definitions. This golden dataset should come from production traffic, not imagination, so it reflects the difficulty distribution that matters.
A scoring function: a rubric-based scorer evaluates each result against task-specific criteria. Factual Q&A may require accuracy, completeness, and no hallucination; summarization may require key-point coverage and no claims absent from the source.
A pass threshold and tracking mechanism: set the minimum score below which the build fails, then preserve benchmark scores over time to reveal quality changes.
Run the harness before and after a change to see whether quality moved in the desired direction. It serves as the equivalent of a green or red test suite.
Regression Testing Without Exact Match
Engineers accustomed to traditional testing must often abandon exact match as the regression standard. Without asserting that output is exactly X, the question becomes how to identify a broken change.
A regression appears when quality metrics fall below a threshold or the score distribution shifts materially. Small random output differences are expected; the important signal is a change in aggregate scores.
If the benchmark pass rate is 87% today versus 91% last week, that indicates a meaningful regression. If it is 87% today versus 88% last week, the difference may reflect ordinary sampling variance, especially with a small dataset. Larger datasets and trend tracking improve the signal.
Some teams use statistical process control charts to separate expected variation from drift. That method fits AI systems because their signal can be noisy. A 3-sigma rule is a reasonable starting point for flagging quality changes.
What Happens Without This Model
Teams that apply traditional testing to LLMs without changing their mental model usually reach one of two outcomes. Tests become brittle, fail during normal variation, and are disabled or loosened until they catch nothing. Or teams omit tests for the LLM layer, leaving user complaints to reveal quality regressions.
Both outcomes stem from the same mistake: treating a distribution-valued system as though it supported point assertions.
Unit tests remain the right choice for deterministic code, so keep them for the scaffolding and add evaluation-based quality checks for the LLM layer. Run both in CI to cover two different failure modes: regressions in written code and changes in the model behavior on which the system depends.
Production Testing as a Third Layer
Pre-deployment evaluation on a golden dataset catches planned changes, but not unplanned production shifts such as provider updates, changing inputs, degraded retrieval, or polluted context from external sources.
Continuous evaluation on production traffic supplies the third layer. It scores a random sample of real calls and alerts when the production distribution diverges from the benchmark baseline. This catches failures that can arise in production without a code change.
Unit tests, benchmark evaluation, and production monitoring together form a coherent quality system. None is enough alone. Without the middle layer's mental model shift, teams remain unable to see important production changes.