A new model gives a clearer answer to a customer asking why an order is late. The answer looks better in a side-by-side review. Then someone opens the run trace: the agent checked inventory at the wrong warehouse and offered a delivery date the shipping system never returned. If the evaluation only grades the final wording, that change passes.
For an agent that works with enterprise systems, a model change is a workflow release. Treat it like one. Test the business result, the evidence behind it, and the actions the agent took along the way. A small set of well-chosen cases can tell you more about release readiness than a large collection of generic prompts.
Consider a hypothetical order-support agent. It can look up an order, retrieve shipment events, check inventory and draft a response. If it cannot resolve the issue, it can prepare a service case for review. The team wants to try a different model to reduce response time without changing those responsibilities.
Start with the everyday case: the order exists, shipment events are current, and the customer is entitled to see them. Then add the cases that take judgment: a split shipment, an order number shared by two business units, a stale carrier scan, a cancelled line and a restricted account. Include a case where the agent has enough information to explain the delay but cannot promise a new date. The expected result there is an honest explanation and a handoff.
Keep the cases compact. Each should have a starting state, a request, the allowed actions and the expected business result. Use synthetic records or approved, redacted examples. Fix the clock when testing a deadline or time window. Give the test environment predictable connector responses so a changing upstream record cannot masquerade as a model regression.
For the split-shipment case, the agent should report the status of each shipment and identify the source of any date it gives. A deterministic check can confirm that the warehouse and shipment IDs in the tool calls match the fixture. It can also verify that no write occurred and that any stated date appears in an authorized source record. Use a reviewer or calibrated model grader for questions that depend on language, such as whether the response clearly distinguishes a confirmed date from an estimate.
Avoid requiring an exact sequence of calls when several routes are valid. One model might fetch all shipment lines in a single request while another uses two narrower reads. If both stay within policy, retrieve the right records and reach the same result, that difference matters mainly for cost and latency. A call to an unauthorized business unit is a different category: it should fail even if the final answer happens to be correct.
OpenAI's agent evaluation guide describes evaluating workflows and using traces to identify where behavior went wrong. OpenTelemetry's trace documentation explains how spans represent individual operations and relate to the broader request. In this example, a trace can help separate a bad model decision from a connector returning stale or incomplete data. Keep routine telemetry focused on operation names, timings, outcomes and opaque record references; do not assume the full prompt or tool payload belongs in a shared logging system.
Run the current and candidate versions against the same fixtures. Keep the prompt, tool definitions and policies fixed for the first comparison. If you change several pieces at once, you can still compare the overall workflows, but you will have a harder time explaining the difference. Repeat the cases where the model has meaningful freedom. A single clean run is weak evidence for a case that sometimes goes wrong.
Set release gates before reading the results. Unauthorized access, unsupported commitments and unintended writes should stop the release. For ordinary quality measures, decide what tradeoff the business can accept. A candidate might be faster but need more handoffs when carrier information is incomplete. That can be reasonable if the handoffs reach the right queue and the work remains manageable. Compare the distribution of response times and the volume of escalations; averages can hide slow or difficult cases.
After the offline checks, use a limited rollout. A read-only comparison can surface cases that fixtures missed. If the production workflow includes writes, make sure the comparison run cannot repeat them. Keep a clear path back to the previous version and check whether failures cluster around a specific connector, business unit or exception type. The goal is to learn which parts of the workflow changed before widening the rollout.
A useful first test suite can start with one common order question, one incomplete record, one permission boundary and one request the agent must decline or escalate. Add cases when operations discovers a new failure. Over time, that collection becomes a practical record of what your business expects the agent to do, even as the model behind it changes.
For adjacent execution controls, see how to prevent duplicate ERP transactions. If you're planning a model change for a workflow that touches business records, Business Dynamics can help define what to test before release.
