Testing an Agent by Replaying Real Tasks in a Sandbox

The demo will pass. Someone types ten tidy requests into a chat window, the agent answers all ten politely, and the support lead signs off. Then, in its first week on a D2C brand's real support queue, the agent changes the delivery address on an order the courier collected the night before, because nobody tested an address change against an order that had already shipped. Testing an agent means replaying real tasks with the system state they arrived with, and checking what the agent left behind.

Why one expected output does not work

A model call that classifies a message can be tested like a function: one input, one expected label. An agent decides which tool to call, in what order, and when to stop. Two correct runs of the same task can look up the order and the customer in a different sequence and still end in the same place. If the test fixes the exact sequence of calls, every small prompt edit breaks every test, and within a month the team stops running them.

So write down what pass means before running anything. Expected: the systems end in the state a competent support executive would have left them in, and the agent did nothing on the way that it was not allowed to do. Actual: whatever the sandbox shows after the run. The path in between is allowed to vary. The end state and a short list of rules about the path are not.

Record real tasks with their state

A task is more than the request text. It is the request plus everything the agent could have looked up at that moment: the order record, the customer's history, the return policy in force that week. Record both. An address change recorded in September and replayed against an order that has since been delivered is a different task, and it tests nothing useful.

Each recorded task becomes a fixture file with the request, the state, and the expected outcome written by someone who knows the process. That is the support lead, not the person who wrote the prompt.

id: addr-change-0142
request: "Pin code galat hai, 560034 kar do please"
state:
  order: { id: OD-88213, status: packed, pincode: "560043" }
  courier_pickup: null
expected:
  order.pincode: "560034"
  unchanged: [order.items, order.amount, refunds]
  reply_contains: ["OD-88213", "560034"]
  escalated: false

Start with twenty tasks from the real queue, with personal data stripped before storage. Cover the common requests and the ones that went wrong. Cases I would want in the first batch:

• An address change on an order still marked packed. Expected: pin code updated, nothing else touched.
• The same change on an order the courier has already collected. Expected: no write, escalation to a person.
• A refund request one day outside the return window.
• "Order kahan hai?" with no order number in the message and two open orders on the account.
• One message asking for an address change and a cancellation at once.
• The ticket that went wrong last month. Replay it, then add it to the regression set.

Sandboxed tools behind the same interface

The agent should not be able to tell it is under test. It calls the same tool names with the same schemas; only the backend differs. In the sandbox, each tool acts on a fresh copy of the recorded state for every task, held in memory or in a throwaway database seeded from the fixture. Writes change the copy, and the checks read the copy afterwards. Calls that would leave the building, such as a WhatsApp message to the customer, a reverse pickup booked with the courier or a refund through the payment gateway, are recorded as intents and never sent.

This is the same tool layer that decides what an agent may read and write in production, with a different implementation behind it. Building the sandbox is mostly a matter of keeping that layer honest. If one tool reaches around it to call the courier API directly, the sandbox cannot catch it, and neither can the permission check in production.

1. 01Recorded taskrequest plus state
2. 02Agent runsprompt under test
3. 03Sandboxed toolsact on a copy
4. 04Checksend state and trace
5. 05Reportpass, fail, trace

Every prompt or model change

The agent code and prompt are the ones that run in production; only the tool backends change.

Three checks, from strict to advisory

Three kinds of check cover most tasks. I keep them in this order of strictness.

1. End state. The pin code on the order equals the one requested. The refund amount matches the policy for that case. No other record changed.
2. Trace rules. No write happened before the read it depends on. Every tool call was in the allowed set. The step count stayed under budget. The agent escalated when the fixture says it should have.
3. Reply content. The order number and the new pin code appear in the reply, and no fact appears that the state does not support. Tone and clarity are judged against a written rubric, by a person or a second model, and that score is a weaker signal than the first two.

• End stateexact; a mismatch fails the task
• Trace rulesexact; a violation fails the task
• Reply factsrequired facts present, none invented
• Wording rubricscored; never fails a task alone

A task passes when every strict check passes. When it fails, the report carries the full trace, so whoever reads it sees what the agent did and in what order, instead of a red cross and a guess.

Regression when the prompt or model changes

Every change to the prompt, the tool descriptions or the model version runs the whole set. Look at which tasks changed status as well as the pass count. A change that fixes two tasks and breaks three has a cost, and the three it broke tell you exactly what that cost is.

Agent runs vary, so run each task more than once. Three runs is a workable start; record how many passed. A task that passes three of three and one that passes one of three are different findings, and a task that flips between runs is telling you something about the prompt. Do not average it away. Keep the prompt, the fixtures and the results in version control together, the same way you would keep prompt versions in Git, so the history of each change is readable a year later.

Once the set passes reliably, the next step is real traffic with no real consequences, which is what shadow runs and canaries are for.

Where agent tests fool you

Testing by chatting. Someone tries a dozen messages, the answers look fine, and nobody can say whether last week's prompt was better than this week's, because nothing was recorded.

Testing against shared staging. Other people change the data, so a failure on Tuesday cannot be reproduced on Wednesday, and the team learns to rerun until green.

Fixing the transcript as the expected output. Every prompt edit then breaks every test, and the tests get switched off.

A set with no production failures in it. The ticket that went wrong is the case nobody wrote down, so the set only contains tasks the agent already handles.

Skipping the escalation cases because they need a person in the loop. Handing a refund dispute to a human is the behaviour that protects the customer, and it is the one never tested. In the sandbox, the escalate tool records an intent like any other external call, so there is no reason to leave it out.

The first twenty tasks

1. Pick one task type, address changes for example, and record twenty real instances with their system state.
2. Have the support lead write the expected outcome for each one.
3. Write the end-state checks and trace rules.
4. Build sandbox backends for the tools that task uses, behind the existing tool layer.
5. Run each task three times, record the pass counts, and commit fixtures, prompt and results together.

The recorded tasks are the part that lasts. Prompts and models will change several times; the fixtures are how you find out what each change did.