Why a model swap can pass the demo and fail old cases

A model substitution changes more than the name in an API request. It can change whether an established support question receives an answer, whether the answer follows a required JSON schema, and whether a concise response preserves all the facts a user needs. A successful demonstration shows that the candidate can handle selected examples; it does not establish that it preserves behavior across the cases the existing system already handles.

The appropriate comparison is the current model and the candidate model on the same frozen cases, with each output judged against the case’s expected behavior. Comparing the two responses only with each other can miss a shared failure, or favor a polished response that violates a requirement.

For illustration, consider GPT-6.1 Sol as the current choice and Claude-Sonnet-5.5 as the candidate. Both appear in the supplied model snapshot associated with AI Plaza, an industry participant with a chat-model catalog. Their assignment to current and candidate roles is hypothetical; it is not a claim that either model performs better. The relevant question is what changes on a particular application’s cases, not which model wins a general contest.

OpenAI’s evaluation guidance recommends task-specific datasets using sources such as production, historical, and human-curated examples, and repeated evaluation as systems change. It distinguishes instruction following, correctness, and formatting among the behaviors that evaluations may need to measure. [1]

Build a frozen regression set around expected behavior

A regression set is a collection of cases used to detect the loss of behavior that matters to an existing application. Start with cases the current model already handles, then add known difficult inputs and examples representing important failure risks. A historical answer is evidence that a case mattered; it is not automatically the correct answer or the desired policy.

Freeze the complete case

Each case should include:

• Input: The user request and the exact system instructions, conversation history, retrieved context, or other material the model is meant to see.
• Expected behavior: The facts, actions, refusal decision, and output format required for that case.
• Grading rules: Separate checks for correctness, refusal behavior, and format validity.
• Importance: Whether a failure is routine, material, or critical to the application.
• Provenance: Where the case came from, with sensitive information removed or protected as appropriate.

For a support case, the expected behavior might require a valid JSON object containing an accurate eligibility decision and a specified explanation. The reference should identify the facts needed to make that decision, not merely provide a preferred writing style.

Freeze the prompts, supplied context, and grading rules before comparing models. Use the same generation settings where both models support them; where settings or provider interfaces differ, record the differences rather than implying exact equivalence. Record model identifiers, run dates, inputs, outputs, and any relevant configuration. If retrieved documents or external tools are part of the task, replay the same captured material where feasible so a changing outside source is not mistaken for a model regression.

Define refusal cases in both directions

Refusal is not a single failure category. A model can refuse a benign request it should answer, or answer a request that the application’s policy requires it to refuse. The regression set needs both kinds of example.

For each case, specify whether the expected behavior is to answer, refuse, or provide an allowed alternative. Judge that decision separately from tone and formatting. A polite refusal can still be inappropriate, and a correctly formatted answer can still violate the case’s safety requirement.

Run the comparison and inspect individual changes

Run both model choices over every frozen case, then grade each response against the reference. The core report is a per-case difference: current pass or fail, candidate pass or fail, and the reason for each result. Include raw outputs so reviewers can inspect what changed.

The following table is illustrative only. It describes possible outcomes, not observed results for either named model.

This comparison also prevents the reverse error: treating every change as a regression. A candidate may newly pass a case, answer more concisely without losing information, or use different wording while meeting every requirement.

Score correctness, refusal, and format independently

A practical case record can contain three separate results:

1. Factual and functional correctness: Are the required facts, conditions, and requested outcome present and supported by the supplied material?
2. Refusal behavior: Does the model answer or refuse as the case requires?
3. Format validity: Is the response parseable and compliant with the required schema, fields, and permitted values?

Not every dimension applies to every case. A free-text answer may have no JSON requirement; a refusal case may have a different completeness standard from an ordinary support answer. Mark dimensions as not applicable when appropriate, rather than forcing an artificial score.

For structured output, distinguish syntactic validity from schema compliance. Parseable JSON can still omit a required field, use the wrong value type, or contain an incorrect decision. Conversely, a correct decision embedded in invalid JSON may be unusable by the downstream application.

Measure length without rewarding it

Record answer length alongside correctness, then review cases where length changes substantially. A shorter answer is not inherently worse: it may remove repetition while preserving every required detail. It becomes a regression when the missing material changes the decision, omits a required condition, or leaves the user without a necessary step.

The same caution applies in reverse. OpenAI notes that automated graders can favor verbosity and recommends validating model-based grading against human judgments. [1] A longer answer should not receive credit merely for sounding comprehensive, particularly if it adds unsupported claims.

Contrast reference-based checks with pairwise judging

Reference-based evaluation and pairwise comparison answer different questions. A reference-based check asks whether each model met the case’s requirements. A pairwise judge asks which response is preferable under specified criteria. Both can be useful, but they are not interchangeable.

Google’s Vertex AI evaluation material describes computation-based and model-based metrics, including comparisons that report candidate and baseline win rates and permit custom criteria. [2] Such comparisons can help reviewers find differences in clarity or completeness across a large set of outputs. They can also surface examples for manual inspection.

A preference score should supplement, not override, explicit requirements. If a judge prefers the candidate’s prose but the application requires valid JSON, the format failure remains. Similarly, if both responses omit a required eligibility condition, a pairwise preference does not make either one correct.

Use deterministic checks where the requirement can be expressed mechanically, such as JSON parsing or required field types. Use human review for ambiguous factual claims, borderline refusals, and cases where a model-based grader disagrees with an explicit check. Review a sample of grader decisions against human labels, and retain disagreements as evidence for improving the evaluation. [1]

Set the release decision before reading the average

A useful swap report separates newly failing cases, newly passing cases, and unchanged failures. It also groups results by behavior: inappropriate refusals, missing required refusals, schema failures, factual errors, and omissions in shorter answers.

Set acceptable limits before the decision, especially for critical failures. For example, a team may require no newly failing cases in a designated safety or legally consequential group while allowing a small number of reviewable style differences elsewhere. Those limits are application decisions, not universal thresholds.

Report both counts and denominators for each group. An overall pass rate can improve while a small, important class of old cases gets worse. A dataset with few examples of a rare failure also provides limited evidence about its frequency. Repeated runs can help identify unstable outputs, but they do not compensate for missing case coverage.

Anthropic’s evaluation guidance describes using high-pass-rate capability tests as continuing regression tests, including when upgrading a model. [3] That supports treating the suite as a maintained operational record rather than a one-time qualification exercise.

Long-term implications for model changes

As models and application requirements change, a frozen comparison remains valuable because it preserves a common point of measurement. It should not, however, become the only evaluation. Add newly discovered failures, changed policy requirements, and representative production cases to a versioned suite; keep the prior frozen set available so results across releases remain interpretable.

The most durable evidence for a model swap is not a demonstration or a single preference score. It is a traceable account of what each model did on the same cases, what the cases required, which failures appeared or disappeared, and which uncertainties remain. That record makes the decision about substitution explicit without turning the exercise into a general model leaderboard.

References

[1] https://developers.openai.com/api/docs/guides/evaluation-best-practices [2] https://cloud.google.com/blog/products/ai-machine-learning/evaluate-ai-models-with-vertex-ai--llm-comparator [3] https://www.anthropic.com/research/demystifying-evals-for-ai-agents