Coverage often tops out at 60 to 70% when teams begin evaluating LLM outputs with a single method. Rubrics are fast and deterministic but miss nuanced reasoning failures. LLM judges capture subtlety but have known biases and cost more at volume. Human review is accurate but cannot scale beyond a few percent of production traffic. None is sufficient alone; the task is assigning each method the cases it handles well and escalating the rest.
Three scoring methods and what each covers
Before combining the methods, clarify what each one measures effectively:
Rubric scoring: Rubrics cover clear, objective criteria well: required fields, expected formats, prohibited phrases, and response length within specified bounds. They are deterministic, inexpensive per call, and stable. They cannot judge nuanced coherence, reasoning accuracy, or tone. A rubric can confirm valid JSON, but not whether its contents answer the question reasonably.
LLM judges: LLM judges evaluate content-dependent criteria such as coherence, instruction adherence, factual plausibility, reasoning completeness, and tone. They can provide a rationale with the score, helping explain failures. They struggle with factual accuracy without a grounding document, consistency on edge cases outside their training distribution, and low-cost operation at high volume.
Human-in-the-loop (HITL): HITL is most accurate for complex judgments and failures that rubrics or judges were not built to detect. Human review also supplies the strongest calibration data for automated methods. It does not scale to continuous production evaluation, because reviewer throughput limits coverage beyond a small traffic sample.
Rubric-based scoring in practice
A well-designed rubric combines weighted binary or ordinal checks. Each check addresses one specific, testable criterion. Reliable examples include:
- Format compliance, does the response parse as valid JSON or follow the specified template structure?
- Required field presence, are all mandatory fields populated with non-empty values?
- Lexical match, does the response include required terminology from a reference set?
- Character or token count within the specified range
- Negative pattern absence, does the response avoid known failure phrases such as 'I cannot answer that' or 'As an AI, I'?
Each check returns a binary result, and weights reflect relative importance. Format failure is usually hard, with weight 1.0 and a zero result regardless of other scores. Lexical matching is softer: missing a preferred term might reduce the score by 0.15 without failing the response.
Rubric scoring's main advantage is complete execution without sampling. With 100,000 responses per day, the rubric evaluates all 100,000. That provides full coverage for its criteria and a population-level view of instruction-following regression over time, with no sampling error and no LLM-call budget.
LLM judges: calibration and common failure modes
Having one language model assess another introduces reliability issues that should be understood before scaling deployment.
The most consistently observed failure modes are:
Verbosity bias: LLM judges often score longer responses higher regardless of quality. A correct two-sentence answer may score below a correct answer padded to six paragraphs. Add explicit conciseness criteria and test response pairs with different lengths but controlled quality.
Sycophancy: Judges often favor confident, fluent answers even when wrong. A confident error can beat a hesitant correct answer, especially in factual accuracy evaluation. Instruct the judge not to reward unsupported fluency or confidence, while recognizing that this will not remove the behavior entirely.
Self-referential inflation: Using the same model family for the evaluated system and judge tends to produce scores above ground truth. This is systematic bias, not random noise. Select a different model family, or at least a different model size, for judging.
Calibration addresses all three issues in practice: create a labeled reference set of at least 75 to 150 examples with human quality scores, run the judge on it, and compute correlation. A judge below 0.65 Pearson correlation with human labels needs prompt revision before production. Because changing inputs degrade this number, recalibrate regularly rather than treating initial calibration as permanent.
Human-in-the-loop: trigger conditions and process design
HITL is more than a fallback for failed automation. It has several active roles in production evaluation:
Escalation for borderline scores: A judge score from 0.55 to 0.70 on a 0 to 1 range, paired with low confidence, indicates that automation cannot commit. This is a defensible human-review trigger. Review is warranted because the case is ambiguous and an error matters, not because automation necessarily failed.
High-stakes query categories: Route responses in categories with meaningful downstream consequences, such as medical information adjacent, legal adjacent, or financial guidance adjacent, to human review regardless of automated score. A false automated pass costs more than the review.
Novel failure patterns: Clusters with unusual characteristics that the automated pipeline cannot confidently classify should receive human review, followed by rubric or judge prompt updates. Novel failures drive improvement in automated methods.
Calibration maintenance: Send a regular sample of all responses to human review, regardless of automated score. This keeps calibration data current and exposes gradual alignment drift that can accumulate over months.
Combining the three layers into one pipeline
The pipeline is sequential rather than three independent tracks. Each layer either produces a final score or passes the response onward:
- Rubric gates first. A hard failure, such as invalid format or a missing required field, receives an immediate fail score and skips the judge. This is more accurate and cheaper. Rubric failures represent 5 to 20% of most production pipelines, so judging an already invalid response wastes money.
- Judge handles rubric passes. Responses passing every rubric check go to the LLM judge on every call or through sampling, depending on budget. The judge assesses coherence, completeness, and factual plausibility.
- HITL receives borderline and high-stakes cases. Borderline judge scores, flagged query categories, and the regular calibration sample proceed to human review.
The pipeline produces one structured evaluation record per response: rubric score and passed or failed checks, judge score and rationale for sampled responses, and human verdict with annotator ID for reviewed responses. Keeping these on the trace instead of separate systems supports incident analysis without cross-database joins.
This architecture still leaves coverage gaps in traffic that is neither sampled nor reviewed. A response that passes rubrics but does not receive judge evaluation gets no quality assessment beyond deterministic checks. At scale, that tradeoff is acceptable because every method cannot cover every response simultaneously. The goal is not zero gaps; it is knowing their location and keeping the failure modes most likely to cause harm out of them. Designing rubric, judge, and HITL criteria around that question is more useful than maximizing aggregate coverage without considering which failures it covers.