Quality

Rehearsals

Rehearsals let a simulated customer talk to your real agent from start to finish, then grade how it went. Use them to test whole conversations — a booking, a cancellation, an angry caller — before your customers have them.

Open Quality → Rehearsals. A rehearsal is a set of tests for one agent.

Spot Checks or Rehearsals?

  • Spot Checks script the caller's exact lines and check each answer. They're fast, and best for "does it still say the right thing?". See Spot Checks.
  • Rehearsals give a simulated customer a goal and a personality, and let it talk to the agent until the goal is met. It answers whatever your agent actually says — it doesn't replay a script. You can run each test many times, in chat or voice, and compare runs.

Rehearsals run on the server, so a run keeps going if you leave the page or close the tab.

Create a rehearsal

Click + Create Rehearsal and fill in:

  • Name and an optional Description — what this rehearsal protects.
  • Target agent — the real agent under test. Runs use its live configuration: prompt, tools, knowledge and workflow.
  • Import this agent's existing spot checks — shown when the agent already has spot checks. Copies them in as tests.
  • Write and run tests automatically — on by default: Callable writes tests from your agent and runs them for you. Turn it off to add and run tests yourself.
  • Testing mode — Chat (text only: fast and cheap, the best way to iterate on prompts, tools and knowledge) or Voice (the full phone pipeline: speech-to-text, text-to-speech, turn-taking, interruptions, silence and voice latency, with a recording).

New rehearsals run in Sandbox: tools that change something outside Callable (bookings, texts, webhooks) aren't executed unless you mock them or choose Live for a run.

The tester

The tester is the simulated customer. Each test gives it a scenario (what it wants and when to stop) and a personality (how it behaves).

  • Basic tester — sensible defaults: English, natural replies.
  • Custom tester — set the model, system instructions, language, first message, max turns and max duration yourself, plus voice and transcriber settings for voice mode.

Pick a Default personality — Friendly, Impatient, Confused, Angry, Skeptical, Talkative, Quiet, Interrupting, Elderly or Professional — or Add personality to write your own. The default is used when a test doesn't pick one.

Tests

Click ✨ Generate Tests and Callable reads the agent's real configuration — prompt, first message, tools, knowledge, workflow, guardrails and transfer rules — and proposes tests across happy paths, variations, missing information, ambiguity, error recovery, edge cases, guardrails and transfers. Add an optional focus, such as cancellations and rescheduling. You review every test before anything is saved.

Or click + Add Test and write one:

  • Full conversation — a simulated customer talks to the agent until the goal is met.
  • Single decision — you write the conversation so far, ending with a customer message, and only the agent's reply to it is graded.

Each test needs at least one success criterion — without one it reports an error, never a pass. Add an AI judge, Text check, Flow check or Structured output criterion, and mark each one required (a failure fails the test) or optional (reported separately). You can also mock a tool's response, so the test never touches your real systems.

Click Regression on a test (Mark as regression test) to add it to your regression set.

Run tests

Click ▶ Run Tests and choose:

  • Tests — All or Selected.
  • Mode — Chat or Voice.
  • Iterations — run each test 1, 5, 10 or 25 times (or a custom number up to 50) to see how consistent the agent is.
  • Environment — Sandbox (side-effect tools aren't executed unless mocked; lookups and transfers still run) or Live (every unmocked tool runs for real against your connected systems). Live asks you to confirm that the run can perform real actions: payments, bookings, messages and record changes.
  • Concurrency — how many tests run at the same time: 1, 5, 10, or a custom number up to 10.

Each voice test is a real call to your agent over a private test line: the tester speaks, the agent answers in its own voice, and the call is recorded. It takes as long as the call does and uses voice and speech-recognition credits like a phone call.

To run only your regression set, open the ⋯ menu and choose Run regression. Results are compared with the previous completed run.

Runs

▶ All runs lists every run across your rehearsals. Open one for the live view: each test's transcript and tool calls appear as they happen, with the pass rate, average latency, cost and errors.

  • Compare with previous — new failures, and any test that changed result or behaviour since the previous run.
  • Consistency across iterations — when tests ran more than once, how consistently each one passed.
  • Cancel run — stops a run that's still going.

Expectations are never adjusted to make a run pass — a changed result means the agent changed.

Results

Open a test's result to read the transcript, the timeline, every tool call, each criterion's verdict and, for a failure, the failure analysis. Voice tests include the recording.

  • Re-grade with current criteria — grades the same conversation against the test's current criteria, without re-running the agent. It's a preview and isn't saved.
  • Rerun test — runs this test again.
  • Create fix task — files the failure as an open issue in Call Watch, with the analysis and a link back to the result.

The agent is never changed automatically.

Who can do what

Admins and developers create, edit and run rehearsals. Members can view them.

Costs

Rehearsals are included on every plan. A run uses AI for your agent and for the tester, and voice tests use voice minutes; all of it is billed like any other usage.

Next steps