What should block a deploy for an LLM application?

An evaluation score becomes operationally important when it can stop a release. Making every metric blocking creates a noisy gate; making none blocking turns evaluation into a report that can be ignored. The boundary should follow the consequence of a false negative and the reliability of the measurement.

A practical gate separates hard checks, advisory checks, and informational measurements. The categories are policy, not properties of a particular evaluation library.

Classify checks by consequence

This section defines the release policy as data. Hard checks cover failures that produce an unsafe action, authorization violation, invalid machine-readable output, or another outcome the team agrees must not ship. Advisory checks cover useful but variable judgments such as tone or explanatory quality. Informational measurements include token use and answer length.

A release gate can represent those outcomes explicitly:

from dataclasses import dataclass
from enum import StrEnum


class Tier(StrEnum):
    HARD = "hard"
    ADVISORY = "advisory"
    INFORMATIONAL = "informational"


@dataclass(frozen=True)
class CheckPolicy:
    name: str
    tier: Tier
    absolute_threshold: float | None
    regression_tolerance: float | None


@dataclass(frozen=True)
class CheckResult:
    name: str
    score: float | None
    status: str
    failed_cases: tuple[str, ...] = ()

Determinism does not automatically earn a veto. A deterministic style rule can still have a low consequence when violated. A model-judged safety check may justify blocking only after calibration shows that its error rate is acceptable for that role.

Compute a verdict from explicit evidence states

This section distinguishes an agent failure from a grader failure and a missing run. Those states require different responses.

from dataclasses import dataclass


@dataclass(frozen=True)
class Decision:
    outcome: str
    reasons: tuple[str, ...]


def decide(
    policy: CheckPolicy,
    result: CheckResult,
    baseline: float | None,
) -> Decision:
    if result.status == "suite_error":
        return Decision("block", (f"{result.name}: suite did not run",))

    if result.status == "unscored":
        outcome = "block" if policy.tier is Tier.HARD else "warn"
        return Decision(outcome, (f"{result.name}: no score available",))

    if policy.tier is Tier.INFORMATIONAL:
        return Decision("pass", ())

    assert result.score is not None
    absolute_failure = (
        policy.absolute_threshold is not None
        and result.score < policy.absolute_threshold
    )
    regression = (
        baseline is not None
        and policy.regression_tolerance is not None
        and baseline - result.score > policy.regression_tolerance
    )

    if absolute_failure or regression:
        outcome = "block" if policy.tier is Tier.HARD else "warn"
        details = result.failed_cases or ("aggregate score regressed",)
        return Decision(outcome, tuple(details))

    return Decision("pass", ())

The treatment of unscored is a product decision. A hard safety grader becoming unavailable may need to block because the required evidence is absent. A single advisory rubric timing out should usually warn. Encode that distinction in policy rather than treating every missing value as zero or as a pass.

Report the value that made the decision

This section makes gate failures actionable. A suite mean can look healthy while one required case falls below its threshold.

If pass/fail is based on individual cases, print the case identifiers, their scores, and their thresholds. If the decision came from a regression rule, print current score, baseline, tolerance, case count, and metric version. Do not display only the aggregate that was easiest to calculate.

Failure messages should avoid user prompts and answers because CI logs often have broad visibility and long retention. Use synthetic case IDs and link authorized reviewers to a protected report when content inspection is necessary.

Calibrate regression tolerance

This section handles the sampling variation of model-judged metrics. A fixed tolerance selected without repeated runs is a guess.

Run the unchanged system several times, estimate the distribution of score differences, and set tolerance above ordinary noise for the intended false-block rate. Keep the case count in the analysis because small suites have less stable means. Bootstrap confidence intervals are often more defensible than a hand-written formula when per-case score distributions are irregular.

Store the calibration date, sample count, judge version, prompt version, and case-set version with the policy. Recalibrate when any of them changes materially.

Absolute case thresholds remain useful alongside regression detection. A stable but unacceptable case should fail even when it matches the baseline; a broad regression can matter even when every case remains just above its minimum.

Protect the baseline

This section treats baseline updates as changes to release policy. A baseline should represent reviewed behavior, not merely the latest run.

Store baseline artifacts in version control or another append-only system. Require an explicit update command that records the source commit, metric versions, case-set hash, reviewer, and reason. Never update a baseline automatically after a successful run, because gradual degradation will redefine itself as normal.

When the metric definition changes, create a new baseline namespace. Scores produced by different rubrics, judge prompts, or aggregation rules are not comparable even when they share a display name.

If no baseline exists, run absolute checks and report regression detection as unavailable. That is weaker evidence, but it is clearer than silently comparing against zero or skipping the gate.

Roll out enforcement with observed data

This section moves a new gate from measurement to enforcement. Begin with local and manual runs to validate cases and reports. Add non-blocking CI execution to measure runtime, cost, flake rate, and missing-score behavior under normal development. Make the status required only after the team has reviewed calibration evidence and agreed on an emergency override process.

An override should expire, require a reason, and remain visible in release history. Track override rate as a health metric; frequent bypasses indicate a policy or measurement problem.

Release gates work when they block on agreed consequences, explain the evidence that triggered the decision, and preserve subjective signals without granting each one a veto. Explicit tiers, calibrated tolerances, protected baselines, and visible missing evidence make the gate strict enough to matter and stable enough to keep enabled.