Keep a Frozen Set of Cases Before Changing the Model

Phase 1: Treat Production Quality as a Change-Detection Problem

A model, prompt, retrieval configuration, or tool schema can change the behavior of an AI system without producing an obvious error. The system may still return fluent answers while omitting a required field, selecting a different tool, or failing to complete an operation. The engineering question is therefore not simply whether the new version appears capable. It is whether a specific system change altered performance on tasks the application needs to handle.

A small, versioned set of real tasks makes that question answerable. Teams rerun the same cases before and after a change, then compare the outcomes against the prior version. The cases should come from actual work: user-reported failures, support requests, bug reports, manual quality checks, and production traces. Anthropic recommends beginning with 20–50 real tasks, each defined by its inputs and success criteria; this is a practical starting range, not a universal sample-size guarantee. [1]

A frozen set does not mean that the evaluation never changes. It means that the cases and scoring rules remain fixed for a particular comparison. If a new failure is added or an existing criterion is revised, the dataset should receive a new version. That distinction preserves the ability to compare runs without quietly changing what “pass” means. Versioned datasets make changes to examples inspectable rather than invisible. [3]

This is different from relying on a broad public benchmark or a handful of manual spot checks. A public score may offer general context, but it does not establish whether a model follows an application’s particular output contract or handles its real edge cases. Manual checks can be useful during exploration, but without stable inputs and criteria they are difficult to reproduce. Evaluation guidance emphasizes defining the objective, dataset, success criteria, and metrics, then comparing system versions over time. [2]

The evaluation target is the complete application behavior, not the model in isolation. A run may depend on the model version, prompt, retrieved material, tool definitions, and execution environment. To interpret a changed result, teams should record which of these components changed. Otherwise, a regression can be detected but not meaningfully attributed.

Phase 2: Build the Regression Set Around Observable Outcomes

Make each case specific enough to repeat

A useful case captures a task the system actually encounters and specifies what a successful result looks like. Depending on the workflow, that may include the input, relevant context, expected output properties, permitted or required tool use, and the final state that should be reached. The case should test a defined behavior rather than invite a general impression of quality.

For example, “handle this customer request correctly” is not yet a scorable task. A more useful case identifies the request, the required action, and the observable result: such as whether the assistant selects the approved lookup tool, supplies a valid account identifier, and produces the expected status. The test can then distinguish a convincing explanation from a completed operation.

Version the cases and the system under test

For each evaluation run, retain the dataset version and the relevant system configuration. That record should identify the model, prompt, retrieval setup, tool schema, and any test-environment details needed to reproduce the comparison. If a team edits an evaluation case, changes its grading rule, or replaces its reference outcome, that should create a new dataset version rather than silently rewriting the old one. Dataset versioning supports repeatable comparisons across experiments. [3]

The same discipline applies to system changes. If the model changes but the prompt and tools remain constant, the comparison is easier to interpret. If several components change together, the evaluation may show that behavior shifted, but it cannot by itself identify which change caused the shift. Where feasible, change one component at a time or record a clear before-and-after configuration.

Score the interaction, not just the final prose

In tool-using workflows, the final answer is only one source of evidence. A system may claim that it completed a task even though the tool call failed or changed the wrong record. Conversely, the interaction may take an unusual path but still produce the correct final state. Evaluation should therefore inspect both the transcript—the sequence of messages and tool calls—and the outcome in the environment. Anthropic’s guidance distinguishes these forms of evidence and recommends checking both for agent evaluations. [1]

A practical scoring plan might separately assess whether a required tool was selected, whether its arguments met the schema, and whether the expected state change occurred. Each criterion should answer a narrow question. For grading language outputs, evaluation guidance favors focused comparisons, classification, or criteria-based scoring over broad, unconstrained judgments; a clear pass/fail rule can make a criterion more consistent. [2]

Not every check needs to be binary. Some tasks involve acceptable variation, such as alternative phrasings that preserve required information. In those cases, criteria should describe what variation is allowed and what must remain true. Repeated trials may also be appropriate because model outputs can vary between runs; multiple trials help distinguish a consistent behavior change from a result observed in a single sample. [1]

Phase 3: Choose an Evaluation Method That Fits the Question

A frozen offline set is most useful for detecting regressions on known tasks before release. Historical production data can extend that view: backtesting runs a proposed version against past interactions, while online evaluation monitors live interactions and may surface new failure patterns. These methods answer related but different questions. The frozen set supports repeatable comparisons; backtesting checks behavior on a broader history; live monitoring can reveal cases the existing set did not anticipate. [4]

For example, suppose a team changes a retrieval strategy. The frozen cases can check whether previously successful questions still receive the required evidence. A backtest can show how the new configuration behaves across a sample of earlier interactions. Online evaluation can then help identify issues arising from current user behavior. None of these replaces the others: a carefully curated set is not a complete representation of production, and production monitoring alone may not provide a stable basis for comparing two versions.

A multi-model workflow illustrates why the comparison must be controlled. A team using AI Plaza, a multi-model AI aggregation platform and active participant in this space, could hold its task set and other relevant configuration constant while comparing system runs involving GPT-6 Sol, Claude-Sonnet-5.5, or Gemini-3.8-Flash. The platform is a workflow context for making such comparisons, not evidence that one model performs better than another. [5] The useful result is the measured behavior on the team’s cases, with the system configuration and scoring criteria recorded—not a generalized claim based on model names.

When comparing methods, the key distinction is what each can establish:

Combining these approaches creates a feedback loop. A production trace that reveals an important failure can be reviewed, made into a scorable case, and added to a new dataset version. The original version remains available for historical comparison. Evaluation systems describe offline evaluation, backtesting, and online evaluation as complementary parts of this broader process. [4]

Phase 4: Reject Vague Cases Before They Distort Decisions

A case is too vague to score when reasonable experts could disagree about whether the output passes, when the grader checks requirements that were never stated, or when no known-good solution demonstrates that the task is achievable under the test conditions.

These are not minor annotation inconveniences. An ambiguous case can make a successful change look like a regression, or a failed change look acceptable. Before adding a case to a blocking evaluation set, ask:

• Can independent reviewers apply the criterion consistently? Anthropic recommends that two domain experts independently agree on pass or fail for a scorable task. If they disagree, clarify the task or its criteria before treating the result as a reliable measurement. [1]
• Does the grader evaluate only stated requirements? If a task asks for a brief summary but the grader penalizes the absence of an unrequested account number, the evaluation is testing an unstated expectation. Amend the task or remove the criterion.
• Can a known-good reference solution pass? A working reference should satisfy the graders. If it cannot, the grading logic, task description, or test environment needs review before a model is judged against it. [1]
• Is the test environment capable of supporting the task? A broken mock service, incorrect fixture, or mismatched schema can create failures unrelated to model behavior. If a frontier model scores zero even across many attempts—including a reported 0% at pass@100—the result is a reason to inspect the task and grader, not proof by itself that the model cannot perform the task. [1]

A short example makes the distinction clear. “Summarize this ticket accurately” leaves room for disagreement about length, emphasis, and which details matter. A more scorable case specifies the information that must be preserved, any prohibited disclosure, and the acceptable output form. The aim is not to force every task into rigid wording; it is to ensure the evaluation measures an explicit requirement rather than an evaluator’s unstated preference.

Phase 5: Use the Set as a Long-Term Observability Asset

A frozen regression set is deliberately incomplete. It preserves known failures as repeatable checks; it cannot anticipate every new query, tool behavior, or shift in user needs. Sampled production traces provide the complementary signal. Offline evaluation can compare versions on curated examples, historical traces can support backtesting, and online evaluation can identify trends or candidates for future test cases. [4]

The set should therefore evolve by documented additions, not by silent edits. When a new failure is validated, add it in a new version with a clear success criterion and a reference outcome where appropriate. Keep prior versions available so that a change in dataset composition is not mistaken for a change in system quality. Dataset versioning supports this distinction by recording edits as new versions. [3]

Over time, this process turns evaluation into a record of what the system is expected to do and what has failed in practice. Before changing a model, prompt, retrieval configuration, or tool schema, rerun the current frozen set. Compare results with the prior version, inspect failures at the transcript and outcome levels, and use production monitoring to discover gaps. The result is not a guarantee that no regression will reach users. It is a more reliable way to notice known quality changes, investigate them, and make the next evaluation set more representative.

References

[1] https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents [2] https://docs.langchain.com/langsmith/manage-datasets [3] https://docs.langchain.com/langsmith/evaluation-types [4] https://developers.openai.com/api/docs/guides/evals [5] https://aiplaza.app