From 6c84ea4d73d3ec476b196f327a935dc25d2bb55b Mon Sep 17 00:00:00 2001 From: Scott Lowe Date: Fri, 4 Sep 2026 16:18:11 -0700 Subject: [PATCH 1/2] docs: rewrite voice agent testing best practices --- fern/assistants/examples/docs-agent.mdx | 134 +++------------- fern/calls/call-ended-reason.mdx | 2 +- fern/calls/troubleshoot-call-errors.mdx | 2 +- fern/changelog/2025-02-25.mdx | 2 +- fern/debugging.mdx | 22 +-- fern/docs-agent-prompt.txt | 16 +- fern/docs.yml | 24 ++- fern/observability/evals-advanced.mdx | 11 +- fern/observability/evals-quickstart.mdx | 5 +- fern/observability/monitoring-quickstart.mdx | 18 ++- fern/observability/simulations-overview.mdx | 15 +- fern/prompting-guide.mdx | 3 + fern/test/chat-testing.mdx | 49 ------ fern/test/evals-best-practices.mdx | 160 +++++++++++++++++++ fern/test/plan-test-coverage.mdx | 125 +++++++++++++++ fern/test/run-and-maintain-tests.mdx | 143 +++++++++++++++++ fern/test/simulations-best-practices.mdx | 144 +++++++++++++++++ fern/test/test-suites.mdx | 133 --------------- fern/test/voice-testing.mdx | 105 +++++++++--- 19 files changed, 754 insertions(+), 359 deletions(-) delete mode 100644 fern/test/chat-testing.mdx create mode 100644 fern/test/evals-best-practices.mdx create mode 100644 fern/test/plan-test-coverage.mdx create mode 100644 fern/test/run-and-maintain-tests.mdx create mode 100644 fern/test/simulations-best-practices.mdx delete mode 100644 fern/test/test-suites.mdx diff --git a/fern/assistants/examples/docs-agent.mdx b/fern/assistants/examples/docs-agent.mdx index 50bc9de79..1cb54c218 100644 --- a/fern/assistants/examples/docs-agent.mdx +++ b/fern/assistants/examples/docs-agent.mdx @@ -517,121 +517,25 @@ Guidelines: - - Create test scenarios to validate your documentation assistant's responses. - - - - 1. Navigate to **Test** > **Voice Test Suites** in your dashboard - 2. Click **Create Test Suite** - 3. Add test scenarios with expected behaviors - 4. Run tests to validate assistant performance - - - ```typescript - import { VapiClient } from "@vapi-ai/server-sdk"; - - const vapi = new VapiClient({ token: "YOUR_VAPI_API_KEY" }); - - // Create test suite for documentation assistant - const testSuite = await vapi.testSuites.create({ - name: "Documentation Assistant Tests", - assistantId: "YOUR_ASSISTANT_ID", - testCases: [ - { - name: "Basic greeting test", - scenario: "User says hello", - expectedBehavior: "Assistant responds with greeting and asks how to help" - }, - { - name: "Documentation query test", - scenario: "User asks about API endpoints", - expectedBehavior: "Assistant uses docsquery tool and provides relevant information" - }, - { - name: "Unknown topic test", - scenario: "User asks about unrelated topic", - expectedBehavior: "Assistant politely redirects to documentation topics" - } - ] - }); - - console.log(`Test suite created with ID: ${testSuite.id}`); - console.log("Next: Go to Dashboard to run the test suite"); - ``` - - - ```python - import requests - - # Create test suite for documentation assistant - url = "https://api.vapi.ai/test-suite" - headers = { - "Authorization": f"Bearer {YOUR_VAPI_API_KEY}", - "Content-Type": "application/json" - } - - data = { - "name": "Documentation Assistant Tests", - "assistantId": "YOUR_ASSISTANT_ID", - "testCases": [ - { - "name": "Basic greeting test", - "scenario": "User says hello", - "expectedBehavior": "Assistant responds with greeting and asks how to help" - }, - { - "name": "Documentation query test", - "scenario": "User asks about API endpoints", - "expectedBehavior": "Assistant uses docsquery tool and provides relevant information" - }, - { - "name": "Unknown topic test", - "scenario": "User asks about unrelated topic", - "expectedBehavior": "Assistant politely redirects to documentation topics" - } - ] - } - - response = requests.post(url, headers=headers, json=data) - test_suite = response.json() - print(f"Test suite created with ID: {test_suite['id']}") - print("Next: Go to Dashboard to run the test suite") - ``` - - - ```bash - curl -X POST https://api.vapi.ai/test-suite \ - -H "Authorization: Bearer YOUR_VAPI_API_KEY" \ - -H "Content-Type: application/json" \ - -d '{ - "name": "Documentation Assistant Tests", - "assistantId": "YOUR_ASSISTANT_ID", - "testCases": [ - { - "name": "Basic greeting test", - "scenario": "User says hello", - "expectedBehavior": "Assistant responds with greeting and asks how to help" - }, - { - "name": "Documentation query test", - "scenario": "User asks about API endpoints", - "expectedBehavior": "Assistant uses docsquery tool and provides relevant information" - }, - { - "name": "Unknown topic test", - "scenario": "User asks about unrelated topic", - "expectedBehavior": "Assistant politely redirects to documentation topics" - } - ] - }' - ``` - - - - - Test suites can only be executed through the Dashboard. Navigate to **Test** > **Voice Test Suites** to run your created tests. - + + Test important decisions and complete conversation outcomes before sharing + the assistant with users. + + + + Check whether the assistant uses the documentation tool, answers from + the result, and redirects requests outside its scope. + + + Run complete conversations with different questions and caller + behaviors, then check whether the user gets a useful answer. + + + + Start with a greeting, a supported documentation question, an ambiguous + question, a tool error, and an unrelated request. See + [testing voice agents](/test/voice-testing) for guidance on building and + maintaining coverage. diff --git a/fern/calls/call-ended-reason.mdx b/fern/calls/call-ended-reason.mdx index 69acb687e..4805db129 100644 --- a/fern/calls/call-ended-reason.mdx +++ b/fern/calls/call-ended-reason.mdx @@ -218,5 +218,5 @@ For SIP trunk setup issues, see [Troubleshoot SIP trunk credential errors](/adva ## Next steps - **[Troubleshoot call errors](/calls/troubleshoot-call-errors):** Step-by-step diagnosis guide organized by what the caller experienced. -- **[Debugging voice agents](/debugging):** General debugging workflow using dashboard tools, logs, and test suites. +- **[Debugging voice agents](/debugging):** General debugging workflow using dashboard tools, logs, Simulations, and Evals. - **[How to report issues](/issue-reporting):** Include your `call_id` and account email when contacting support. diff --git a/fern/calls/troubleshoot-call-errors.mdx b/fern/calls/troubleshoot-call-errors.mdx index bdbba3efd..f4c8c1b00 100644 --- a/fern/calls/troubleshoot-call-errors.mdx +++ b/fern/calls/troubleshoot-call-errors.mdx @@ -266,7 +266,7 @@ These are not errors — they indicate the call ended as expected. ## Next steps - **[Call end reasons](/calls/call-ended-reason):** Complete reference of every `endedReason` code. -- **[Debugging voice agents](/debugging):** General debugging workflow using dashboard tools, logs, and test suites. +- **[Debugging voice agents](/debugging):** General debugging workflow using dashboard tools, logs, Simulations, and Evals. - **[Debug forwarding drops](/phone-calling/in-call-control/transfer-calls/debug-forwarding-drops):** Deep dive into transfer failures. - **[Troubleshoot SIP trunk errors](/advanced/sip/troubleshoot-sip-trunk-credential-errors):** Resolve SIP credential validation failures. - **[Troubleshoot SIP response codes](/advanced/sip/troubleshoot-sip-response-codes):** Identify where a SIP request failed and what to check next. diff --git a/fern/changelog/2025-02-25.mdx b/fern/changelog/2025-02-25.mdx index be8964c3d..d6e066a14 100644 --- a/fern/changelog/2025-02-25.mdx +++ b/fern/changelog/2025-02-25.mdx @@ -1,6 +1,6 @@ ## Test Suite APIs, Enhanced Call Transfers, Voice Model Enhancements -1. **Introducing Test Suite Management APIs:** You can now test your assistant conversations before deploying them by creating [end-to-end tests](https://docs.vapi.ai/test/voice-testing#step-1-create-a-new-test-suite), [adding test cases](https://docs.vapi.ai/test/voice-testing#step-3-add-test-cases), and [running and reviewing test suites](https://docs.vapi.ai/test/voice-testing#step-5-run-and-review-tests). You can configure these tests through the [Test Suites dashboard page](https://dashboard.vapi.ai/test-suites) and [Test Suite APIs](https://docs.vapi.ai/api-reference/test-suites/test-suite-controller-find-all-paginated), and learn more in the [docs](https://docs.vapi.ai/test/voice-testing). +1. **Introducing Test Suite Management APIs:** You can now create, run, and review end-to-end assistant tests through the Test Suites dashboard and APIs. Test Suites are now deprecated; use [Simulations](/observability/simulations-overview) for current testing guidance. Test Suite Management APIs diff --git a/fern/debugging.mdx b/fern/debugging.mdx index 3fdc5dabe..2a88010d5 100644 --- a/fern/debugging.mdx +++ b/fern/debugging.mdx @@ -58,9 +58,12 @@ Start with these immediate checks before diving deeper: Use [dashboard](https://dashboard.vapi.ai/) testing features: - - - Automated testing for assistants + + + Test outcomes across complete conversations + + + Test decisions at specific conversation points Test tools with sample data @@ -132,15 +135,14 @@ vapi listen --forward-to localhost:3000/webhook `vapi listen` is a local forwarder that requires a separate tunneling service. Update your webhook URLs in Vapi to use the tunnel's public URL. [Learn more →](/cli/webhook) -### Voice Test Suites +### Simulations and Evals -Navigate to `Test > Voice Test Suites` to: -- Run automated tests on your assistants -- Test conversation flows with predefined scenarios -- Verify assistant behavior across different inputs -- Monitor performance over time +Use **Simulations** to reproduce a problem across a complete conversation and +check the final outcome. Use **Evals** to isolate a specific decision, such as +choosing a tool, asking for missing information, or escalating the call. - + + + Test outcomes across complete conversations + + + + Plan testing coverage before and after launch diff --git a/fern/observability/simulations-overview.mdx b/fern/observability/simulations-overview.mdx index 7758459fa..45f4e670c 100644 --- a/fern/observability/simulations-overview.mdx +++ b/fern/observability/simulations-overview.mdx @@ -46,7 +46,9 @@ Simulations support two conversation modes: -Both modes run the same AI tester conversation and score it the same way. Chat mode simply removes the audio layer: the AI tester and the [**assistant**](/assistants) or [**squad**](/squads) exchange text directly instead of synthesized speech and transcription. +Both modes run the same scenario. Chat mode removes the audio layer: the AI +tester and the [**assistant**](/assistants) or [**squad**](/squads) exchange text +directly instead of synthesized speech and transcription. Use **chat mode** for rapid iteration during development, then switch to **voice mode** for final validation. @@ -62,11 +64,13 @@ Use **Simulations** to confirm that an [**assistant**](/assistants) or [**squad* - **Regression testing**: After changing a prompt, tool, or configuration, rerun a suite to catch behavior that broke. - **Edge cases and difficult behavior**: Give the AI tester different personalities to see how the assistant or squad handles impatient, confused, or interrupting behavior. - **Guardrail validation**: Confirm the assistant or squad stays within its boundaries, such as never promising a refund it can't authorize, and keep the results as evidence. -- **Tool-call validation**: Confirm the assistant or squad calls tools with the right arguments and handles failures gracefully, such as surfacing a booking error instead of falsely confirming it. +- **Tool-backed outcomes**: Confirm the assistant or squad completes a workflow and handles tool failures gracefully, such as surfacing a booking error instead of falsely confirming it. Use [**Evals**](/test/evals-best-practices) for exact tool and argument checks. ## Simulations compared with Evals -Use [**Evals**](/observability/evals-quickstart) to test the logic of an [**assistant**](/assistants) or [**squad**](/squads). Use **Simulations** to test how that assistant or squad handles a real conversation. Both are testing tools in Vapi, and they complement each other. +Use [**Evals**](/observability/evals-quickstart) to test decisions at specific +points in a conversation. Use **Simulations** to test outcomes across a complete +conversation. Both are testing tools in Vapi, and they complement each other. Evals verify specific, expected behavior. You script a mock conversation with user turns and expected responses, then check the result with exact match, regex, an AI judge, or tool-call checks. Evals are precise and repeatable, which makes them a good fit for unit-style checks. @@ -74,7 +78,7 @@ Simulations test live interactions. An AI tester with a defined personality and | Comparison | Evals | Simulations | | -- | -- | -- | -| **What you're testing** | Assistant or squad logic | How an assistant or squad handles a real conversation | +| **What you're testing** | A decision at a specific conversation point | The outcome of a complete conversation | | **What you provide** | A scripted conversation with expected responses | An AI tester personality and an intent | | **How it runs** | Repeatable, turn-by-turn checks | Dynamic; the AI tester improvises the conversation | | **Transport** | Chat (mock conversations) | Voice, or text-only | @@ -85,6 +89,9 @@ Choose Evals when you want to lock down a specific response or verify a tool cal Choose Simulations when you want to see how your assistant or squad performs against an AI tester with different personalities, multi-turn flows, squad handoffs, or voice. +See [testing voice agents](/test/voice-testing) for guidance on combining both +tools into one coverage plan. + ## Next steps diff --git a/fern/prompting-guide.mdx b/fern/prompting-guide.mdx index 726135898..b5bac7b68 100644 --- a/fern/prompting-guide.mdx +++ b/fern/prompting-guide.mdx @@ -40,6 +40,9 @@ Your success rate is the percentage of requests your agent handles from start to Validate prompt changes against a representative test set, not single calls. Probabilistic regressions don't show up in one-off testing — they only become visible across many iterations. +Use [plan test coverage](/test/plan-test-coverage) to cover both complete +conversation outcomes and important decisions within those conversations. + ## The process Follow a structured approach to prompt engineering: diff --git a/fern/test/chat-testing.mdx b/fern/test/chat-testing.mdx deleted file mode 100644 index 29a7c15d6..000000000 --- a/fern/test/chat-testing.mdx +++ /dev/null @@ -1,49 +0,0 @@ ---- -title: Chat Testing -subtitle: Automated text-based testing for AI agents -slug: /test/chat-testing ---- - -## Overview - -Chat Test Suites allow you to evaluate your AI agents through simulated text conversations. This is our recommended solution for testing as it is much faster than voice testing and lets you isolate testing the behavior of your agent. - -## How Chat Testing Works - -1. **Simulation:** Our AI tester engages with your agent in a text-based conversation. -2. **Scripted Interaction:** The testing agent follows your predefined script to simulate specific customer scenarios. -3. **Transcript Capture:** The conversation is captured as a transcript. -4. **Evaluation:** A language model (LLM) assesses the transcript against your success criteria. - -## Designing your tests - -Good test design is critical to evaluating your agent. You'll want to consider testing: - -1. The tool calls of your agent. Set your script to schedule an appointment or call a transfer tool. At the evaluation step, your rubric will have context of the tool call history to evaluate success. -2. Knowledge base integrations. Test different Q&A to make sure that your agent responds as expected. -3. Legal / compliance issues. Ask the agent to answer things it's not supposed to, and verify that it refuses to answer. -4. Personality. Simulate an angry, frustrated or manipulative customer, and make sure your assistant handles the situation well. - -## Benefits of Chat Testing - -- **Speed:** Chat tests execute faster than voice tests, allowing for rapid iteration. -- **Cost-Effective:** No TTS or STT models are used during chat testing. -- **Focused Assessment:** Evaluate pure conversational ability without audio-related variables. -- **Higher Test Volume:** Run more tests in less time to ensure comprehensive coverage. - -## Creating Chat Tests - -You can create chat tests as part of a Test Suite: - -1. Navigate to the **Test** tab and select **Test Suites**. -2. Create a new Test Suite or edit an existing one. -3. When adding tests, select **Chat** as the test type. -4. Define your script and success criteria as detailed in the [Test Suites](./test-suites) documentation. - -## Best Practices for Chat Testing - -- Use chat tests for rapid iteration during development. -- Create variations of the same scenario to test different user inputs. -- Test edge cases and potential misunderstandings. - -For comprehensive instructions on creating and managing test suites that include chat tests, refer to the [Test Suites](./test-suites) documentation. diff --git a/fern/test/evals-best-practices.mdx b/fern/test/evals-best-practices.mdx new file mode 100644 index 000000000..961fa8641 --- /dev/null +++ b/fern/test/evals-best-practices.mdx @@ -0,0 +1,160 @@ +--- +title: Test decisions with Evals +subtitle: Learn to check the next decision your voice agent makes at a specific point in a conversation. +description: "Learn how to choose Eval checkpoints, cover decision boundaries, and select the right judge for each voice agent behavior." +slug: /test/evals-best-practices +--- + +## Overview + +Evals answer: **At this exact moment, does the agent make the right next decision?** + +An Eval provides a mock conversation state, lets your assistant or squad respond at selected checkpoints, and judges the result. Use Evals for decisions such as whether to ask a question, call a tool, transfer a caller, refuse a request, or give a specific kind of answer. + +Evals are path dependent: the messages before a checkpoint establish the exact situation being tested. Use [**Test outcomes with Simulations**](/test/simulations-best-practices) when you care about the result of a complete conversation and want the caller and agent to reach it through different valid paths. + + + Evals run mock conversations at the text and model layer. They do not test the voice, transcriber, audio quality, or turn-taking behavior. Use a voice Simulation for those checks. + + +## Test decisions that matter + +Prioritize decisions that are frequent, hard to reverse, or costly when wrong: + +- **Act or ask:** Does the agent have enough information to complete the task, or should it ask a follow-up question? +- **Choose a tool:** Does the agent select the right tool and send the right values? +- **Proceed or stop:** Does the agent meet identity, consent, eligibility, or policy requirements before acting? +- **Handle a result:** Does the agent react correctly when a tool succeeds, returns no options, or fails? +- **Transfer or continue:** Does the agent route the caller at the right time and to the right destination? +- **Say or withhold:** Does the agent provide allowed information and avoid unsupported or restricted claims? + +Start with a narrow question. "Does the agent handle appointment booking?" is too broad for one checkpoint. "After the caller provides all required details, does the agent call the booking tool with the agreed time?" is specific enough to diagnose. + +## Provide only the context the decision needs + +Build the mock conversation up to the moment just before the decision. Include relevant user, assistant, system, and tool messages so the agent sees the same facts and constraints it would have at that point in a real call. + +Include: + +- Facts already provided by the caller. +- Questions the agent already asked. +- Applicable policy or system instructions. +- Tool results the agent must interpret. +- Any unresolved ambiguity that should affect the next action. + +Remove unrelated turns. Extra context makes the reason for a failure harder to understand and can accidentally change the behavior under test. + +## Test both sides of the boundary + +For every important action, test when it **should** happen and when it **should not**. This prevents a prompt change from fixing missed actions by triggering the action everywhere. + +For an appointment-booking agent, a useful group of Evals looks like this: + +| Conversation state | Expected next decision | +| --- | --- | +| Identity, date, time, and timezone are known; the slot is available. | Call the booking tool with the agreed details. | +| The caller gave a date and time but no timezone. | Ask for the timezone; do not book yet. | +| The caller has not passed required identity verification. | Request verification; do not book yet. | +| The booking tool reports success with a confirmation number. | Confirm the booking and share the final details. | +| The booking tool reports an error. | Explain that the booking is not confirmed and offer a safe next step. | + +Add cases close to the boundary, not only obvious examples. Ambiguous dates, partial account details, similar tool names, and conflicting requests often reveal more than another straightforward happy path. + +## Choose the simplest judge that fits + +Evals support exact matching, pattern matching with regular expressions (regex), +and AI judges. Exact checks can also validate tool calls and their arguments. + +| Judge | Use it when | Avoid it when | +| --- | --- | --- | +| **Exact match** | The precise tool name, number of calls, argument keys, and values matter. | Several different responses would all be valid. | +| **Pattern match (regex)** | A response must contain a stable phrase, identifier, or format but other wording may vary. | You need to judge meaning, correctness, or tone. | +| **AI judge** | The decision can be expressed in several valid ways or requires semantic judgment. | A tool-call or format check can prove the requirement. | + +Use exact response text only when the wording itself is a requirement, such as a mandated disclosure. For most customer-facing replies, judge the meaning so harmless wording changes do not break the test. + +Give an AI judge one clear pass/fail question. For example: + +```text title="AI judge criterion" wordWrap +Pass only if the assistant says the appointment is not yet confirmed and gives +the caller a clear next step. Fail if it implies that the booking succeeded. +``` + +Avoid broad criteria such as "The response was good." If you care about correctness, policy compliance, and tone, use separate checkpoints or checks so the failure remains actionable. + +Test an AI judge against responses that informed reviewers agree should pass and +fail. If the judge disagrees, refine its criterion before using it as a release +check. + +## Keep each Eval focused + +A mock conversation can contain more than one checkpoint, but each checkpoint should test one decision. Give the Eval a name that states the condition and expected behavior, such as: + +- `asks for timezone before booking` +- `does not refund an ineligible order` +- `transfers urgent safety concern to specialist` +- `does not confirm after booking tool error` + +Focused Evals are easier to review, reuse, and update. They also make it clear whether a change improved one decision while breaking another. + +## Turn real failures into regression tests + +When a production call contains a wrong decision, use the relevant messages to create an Eval. End the mock context immediately before the mistake, then define the correct next action. + +After the first Eval reproduces the failure: + + + + Add a nearby case where the opposite decision is correct. + + + Confirm the judge accepts a known-good response. + + + Fix the assistant and rerun both cases. + + + Keep the Evals so the same issue cannot return unnoticed. + + + +Do not copy the entire call unless every turn is needed. Preserve the facts that caused the decision and remove customer data that the test does not require. + +## Review failures before changing the agent + +A failed Eval can mean the agent made the wrong decision, but it can also mean the mock context was incomplete or the judge was too strict. Review the actual response and ask: + +- Did the agent have all the information needed to pass? +- Would two reviewers agree with the expected decision? +- Did an exact or regex check reject a valid response? +- Does the AI judge describe one objective clearly? + +Fix the test when it does not represent the intended situation. Fix the agent when the failure is fair. Keep the expected decision stable unless the product or policy requirement changes. + +## Avoid common pitfalls + +| Pitfall | Better approach | +| --- | --- | +| Using an Eval to prescribe an entire successful conversation | Use a Simulation for the end-to-end outcome. | +| Testing only when an action should happen | Add a paired case where the agent must wait, ask, refuse, or choose another action. | +| Exact-matching ordinary conversation | Use regex for stable formats or an AI judge for meaning. | +| Giving an AI judge several vague goals | Use one explicit, binary criterion at a time. | +| Omitting prior tool results or caller details | Provide the state that makes the expected decision possible. | +| Treating a passing Eval as proof of voice quality | Validate the same critical flow with a voice Simulation. | + +## Next steps + + + + Create mock conversations, checkpoints, and judges. + + + Explore complex messages, tools, runs, and automation. + + + Prioritize customer outcomes, critical decisions, and high-risk behavior. + + + Turn failures into regression coverage and keep tests current. + + diff --git a/fern/test/plan-test-coverage.mdx b/fern/test/plan-test-coverage.mdx new file mode 100644 index 000000000..b28288cc9 --- /dev/null +++ b/fern/test/plan-test-coverage.mdx @@ -0,0 +1,125 @@ +--- +title: Plan test coverage +subtitle: Learn to cover the conversations and risks that matter most +description: Plan practical voice agent test coverage across customer outcomes, agent decisions, failure paths, guardrails, and voice behavior. +slug: /test/plan-test-coverage +--- + +## Overview + +Good coverage does not mean scripting every sentence a caller might say. It means having enough evidence that your agent handles the goals, decisions, and risks that matter to your business. + +Start with a focused set of high-value tests. Add coverage as you learn from manual testing, support issues, and production conversations. + +## Start with clear requirements + +Write down what the caller should be able to accomplish and what the agent must never do. Use language that a support operator, product manager, and subject-matter expert can judge the same way. + +| Too vague | Clear enough to test | +| --- | --- | +| The agent is helpful. | If the requested time is unavailable, the agent offers available alternatives. | +| The agent protects customer data. | The agent does not share appointment details until the caller verifies their identity. | +| The agent handles transfers well. | The agent transfers billing disputes to Billing and explains the handoff before transferring. | + + + A requirement is ready to test when two informed teammates would independently reach the same pass or fail decision. + + +## Prioritize by risk + +You do not need hundreds of tests to get started. Test these areas first: + +- **High impact:** A failure could expose private data, move money, make an unauthorized change, miss an urgent escalation, or create legal risk. +- **High frequency:** Callers use the behavior often or depend on it to complete the agent's main job. +- **High uncertainty:** The behavior is new, recently changed, inconsistent, or dependent on an external tool. + +Cover high-impact behavior even when it is rare. Then cover the most common customer goals and the areas that change most often. + +## Build coverage in layers + +Use this checklist for each important customer goal. + +| Coverage area | What to include | Start with | +| --- | --- | --- | +| **Core outcome** | The caller completes the goal and receives an accurate confirmation or next step. | Simulation | +| **Critical decisions** | The agent asks the required question, chooses the right tool or route, and uses the right information. | Eval | +| **Both directions** | The action happens when it should and does not happen when it should not. | Evals | +| **Missing or unclear information** | The agent asks for what it needs instead of guessing. | Eval, then Simulation | +| **Tool and service failures** | The agent explains the problem, avoids a false confirmation, and offers a safe next step. | Eval, then Simulation | +| **Guardrails** | The agent protects private data, follows policy, refuses unsafe requests, and escalates when required. | Evals and Simulations | +| **Caller variation** | Clear, vague, impatient, confused, or changing requests still lead to an appropriate result. | Simulations | +| **Handoffs** | The agent chooses the right destination, gives context, and handles an unavailable destination. | Evals and Simulations | +| **Voice experience** | Important names, dates, numbers, and domain terms are understood and spoken clearly; pacing and turn-taking work when you review the recording. | Voice Simulations | +| **Known failures** | Every important fixed issue has a test that recreates the original conditions. | Whichever tool reproduces it best | + +Many requirements need both tools. Use an Eval to protect a critical choice and a Simulation to prove that the full conversation still succeeds. + +## Cover both success and restraint + +Testing only the action you want can create a new problem: an agent that takes that action too often. Pair positive cases with negative cases. + +For an identity check, test that the agent: + +- Shares account details after successful verification +- Does not share account details after failed verification +- Asks for missing verification information +- Escalates or ends safely after repeated failed attempts + +Use the same pattern for refunds, discounts, appointments, transfers, cancellations, and other consequential actions. + +## Example coverage map + +For an appointment scheduling agent, a useful starting map might look like this: + +| Requirement | Decision coverage with Evals | Outcome coverage with Simulations | +| --- | --- | --- | +| Book an available time | Use the booking tool with the correct date, time, and customer | The booking tool reports success and the caller hears the correct confirmation | +| Protect appointment details | Share details only after successful verification | An unverified caller cannot obtain private appointment information | +| Handle an unavailable time | Do not claim success; ask the caller to choose an alternative | The caller chooses an available alternative and completes the booking | +| Recover from a tool failure | Do not invent a confirmation number | The caller receives an honest explanation and a safe next step | +| Support changes of mind | Choose the new action instead of continuing the old one | A caller can switch from booking to canceling without an unintended booking | + +This map describes behavior, not exact wording. An agent should not fail because it used a different natural phrase or took a different valid route. + +## Set a launch bar + +Before a voice agent handles production traffic, confirm that: + +- Every primary customer goal has reusable end-to-end Simulation coverage +- Every high-risk decision has cases for when the action should and should not occur +- Each primary goal includes at least one likely failure or recovery case +- Critical handoffs and guardrails have dedicated coverage +- Primary and high-risk outcomes pass repeatedly, with the number of runs based on the impact and variability of the behavior +- Critical conversations have been tested in voice mode +- A person has reviewed representative transcripts and recordings +- Tools that can make real changes use safe test data or [mock responses](/observability/simulations-advanced#mock-tool-responses) + +Treat this as a minimum, not a finish line. Increase coverage for regulated, high-volume, or high-consequence use cases. + +## Keep coverage current + +Test coverage is a living record of what your team expects. Update it when: + +- A customer or support issue reveals a new failure mode +- You add a customer goal, tool, handoff, language, or policy +- You change a prompt, model, voice, transcriber, or provider +- A business requirement changes + +When a test fails, first decide whether the agent is wrong or the expectation has changed. Do not weaken a test only to make it pass. Review the transcript, correct the agent or the requirement, and keep the test as regression coverage. + +## Next steps + + + + Review how Evals and Simulations work together. + + + Turn customer goals into complete conversation tests. + + + Create focused checks for critical choices and guardrails. + + + Run the right tests before release and learn from production failures. + + diff --git a/fern/test/run-and-maintain-tests.mdx b/fern/test/run-and-maintain-tests.mdx new file mode 100644 index 000000000..1283d8f51 --- /dev/null +++ b/fern/test/run-and-maintain-tests.mdx @@ -0,0 +1,143 @@ +--- +title: Run and maintain tests +subtitle: Learn to run tests on a useful cadence, investigate failures, and keep coverage current. +description: Build a repeatable release and incident workflow with Evals, Simulations, call review, and production monitoring. +slug: /test/run-and-maintain-tests +--- + +## Overview + +Tests reduce risk only when your team runs them regularly, investigates failures, and keeps them aligned with the current assistant. Treat your tests as a living record of the decisions and outcomes customers should be able to expect. + +## Separate improvement tests from regression tests + +Use **improvement tests** to explore difficult behavior the agent cannot handle +reliably yet. These tests show where to improve, so they do not all need to pass +before every release. + +Use **regression tests** to protect behavior the agent already needs to perform. +These tests should remain consistently passing. When the agent becomes reliable +on an improvement test that matters in production, keep it as regression +coverage. + +## Use a repeatable cadence + +Run the smallest useful set while you work, then increase coverage as a change gets closer to customers. + +| When | What to run | What good looks like | +| --- | --- | --- | +| While designing or changing the assistant | Run affected Evals for quick feedback. Follow with Chat Simulations for the journeys the change could affect. | The intended decision works and related conversations still reach the right outcome. | +| Before a release | Run every test for your critical journeys. Repeat Simulations and include Voice Simulations for customer-facing voice paths. | Results are consistent, with no unexplained failure on a release-blocking requirement. | +| Immediately after a release | Place a small set of controlled calls through the real phone and tool path. Review their artifacts. | The deployed version behaves as tested, including its audio and integrations. | +| In production | Review a sample of real calls, respond to [monitoring alerts](/observability/monitoring-quickstart), and add new customer patterns to coverage. | New failures become regression tests instead of recurring incidents. | + +Rerun affected tests whenever you change prompts, tools, models, voices, transcribers, call settings, or business rules. A result from an older assistant version doesn't validate the current version. + +## Repeat Simulations + +AI conversations can take different paths from the same starting point. One passing Simulation shows that one path worked, not that the journey is reliable. + +- Use one iteration while you build or debug a Simulation. +- Repeat release checks based on the impact of failure and how much the result has varied before. +- Use more iterations for critical journeys and intermittent failures when the extra evidence justifies the time and cost. + +Compare the iterations instead of averaging them away. If one of five calls violates a critical requirement, treat the behavior as unstable until you understand why. See [reviewing and rerunning Simulation results](/observability/simulations-manage#review-run-results). + +## Review more than the pass or fail label + +Review every failure and sample some passes. An automated evaluator can miss a problem you didn't include in its success criteria. + +| Review | Look for | +| --- | --- | +| Transcript | The first unexpected decision, a missing question, an incorrect tool call, a confusing response, or a conversation that reached the wrong outcome. | +| Recording | Misheard words, poor pronunciation, clipped speech, awkward pauses, interruption handling, and turn-taking problems. | + +For a Voice Simulation or controlled call, read the transcript and listen to the recording when it's available. The transcript helps you inspect what the assistant decided. The recording tells you what the caller actually experienced. Learn how to [view call transcripts and recordings](/assistants/call-recording#accessing-artifacts). + +Periodically check automated criteria and AI judges against conversations that +informed reviewers agree should pass or fail. If the automated result disagrees, +revise the criterion or judge before changing the agent. + +## Triage failures consistently + +Start with the first point where the test and expected behavior diverge. A later error is often only a symptom. + + + + Check that the scenario is realistic, the expected behavior hasn't changed, + and the success criteria measure something observable. + + + Inspect the transcript, evaluation result, tool activity, and recording when + audio is involved. + + + Use the signals below to choose the next owner and test. + + + Rerun the failing test, then run nearby tests that could be affected by the + same change. + + + +| Signal | Likely next step | +| --- | --- | +| Wrong reply, next action, tool choice, or tool arguments at a known point | Isolate the decision with an Eval. Check the prompt, tool definition, and conversation state. | +| The conversation takes an unexpected route or misses its final goal | Reproduce the full journey with a Simulation. Check the scenario, assistant instructions, and tool behavior. | +| The transcript doesn't match what was spoken | Use a Voice Simulation and recording to investigate the transcriber and audio input. | +| Speech is clipped, slow, hard to understand, or handles interruptions poorly | Use a Voice Simulation, then verify with a controlled real call. Check voice and turn-taking settings. | +| A request errors or times out | Inspect the call and tool logs. Follow the [debugging guide](/debugging) before changing the test. | + +Don't loosen a success criterion or rerun until you get a pass without explaining the failure. Record intermittent failures so the team can track whether a change made them better or worse. + +## Turn production issues into regression tests + +After you fix a production issue, add the smallest test that would have caught it before release. + +| Production issue | Regression coverage | +| --- | --- | +| The assistant made the wrong decision at a specific point | Create an [Eval](/observability/evals-quickstart) with the relevant conversation state and expected next response or tool call. | +| The call failed to reach the right result across several turns | Create a [Simulation](/observability/simulations-quickstart) for that goal and customer behavior. | +| The problem depended on speech, interruptions, or the phone path | Add a Voice Simulation and keep a controlled real-call check. Monitor the issue in production if automation can't reproduce it reliably. | + +Remove personal information from production examples. Use synthetic values and safe test accounts. Mock or disable any tool action that could charge a card, contact a real customer, or make another irreversible change. + +## Give every critical journey an owner + +Assign one person to each business-critical journey. That owner keeps the expected outcome current and reviews failures. In your team tracker, also record: + +- which tests cover the journey +- when those tests must run +- which failures block a release +- who investigates assistant, tool, or platform problems + +Update the tests in the same change whenever the business requirement changes. If your team accepts a known failure, document who approved it, why it is safe, and when it will be reviewed again. + +## Keep real-call checks and monitoring + +Automated tests can't reproduce every carrier route, device, background sound, speaking style, live dependency, or real customer reaction. They also judge only the requirements you define. + +Before launch, and after meaningful voice or telephony changes, place controlled calls through the number and integrations customers will use. Use test data, verify inbound and outbound paths that matter, and follow your recording consent and data-handling requirements. + +After launch, use [Monitoring](/observability/monitoring-quickstart) to detect changes across real calls and inspect [call artifacts](/assistants/retrieve-call-artifacts) when you investigate. Automated tests show what could work before release. Real-call review and monitoring show what is working now. + + + Don't let a controlled test call trigger production payments, messages, bookings, or other irreversible actions. Use sandbox integrations or dedicated test records. + + +## Next steps + + + + Prioritize customer outcomes, critical decisions, and high-risk behavior. + + + Review results, rerun suites, and update Simulation coverage. + + + Create and run focused decision checks. + + + Detect changes and failures across production calls. + + diff --git a/fern/test/simulations-best-practices.mdx b/fern/test/simulations-best-practices.mdx new file mode 100644 index 000000000..5f85341a9 --- /dev/null +++ b/fern/test/simulations-best-practices.mdx @@ -0,0 +1,144 @@ +--- +title: Test outcomes with Simulations +subtitle: Learn to design full-conversation tests around the result your voice agent should achieve. +description: "Learn how to write realistic simulation scenarios, useful success criteria, and durable coverage for voice agents." +slug: /test/simulations-best-practices +--- + +## Overview + +Simulations answer: **Can the agent guide the caller to the right result?** + +An AI tester follows a goal and adapts during a complete conversation with your assistant or squad. Score the final result and important guardrails, not one exact route through the conversation. This keeps the test useful when the caller phrases something differently or the agent asks an unexpected but reasonable question. + +Use [**Test decisions with Evals**](/test/evals-best-practices) when you need to test the agent's next decision at a specific point in a conversation. + +## Start with the customer outcome + +Write down what must be true when the conversation ends. Use evidence that distinguishes a completed task from a confident-sounding promise. + +For an appointment change, avoid a test that requires a fixed sequence: greet the caller, ask for their name, offer Tuesday at 3:00 PM, call the scheduling tool, then read a confirmation. + +Instead, test the durable requirements: + +- The appointment is changed to the time the caller accepts. +- The agent confirms the change only after the scheduling result reports success. +- The agent gives the caller the final date and time clearly. +- If the change fails, the agent explains the next safe option instead of claiming success. + +The agent can then recover from clarifications or conversational detours without failing a valid test. Require a specific step only when the step is itself a business, safety, or compliance requirement. + +## Give the AI tester a goal, not a script + +The **scenario** should describe the caller's goal and relevant facts. The **personality** should describe how the caller behaves. + +| Include in the scenario | Include in the personality | +| --- | --- | +| What the caller wants | Tone and patience | +| Facts the caller knows | How concise or talkative they are | +| Information they may provide | Whether they interrupt or ask follow-up questions | +| Constraints and acceptable alternatives | How they respond to unclear answers | +| When to end the conversation | Voice, transcriber, and turn-taking settings | + +For example: + +```text title="Scenario" wordWrap +You need to move your appointment because of a conflict. Ask for Tuesday at +3:00 PM. If that time is unavailable, accept Wednesday at 10:00 AM. Provide +your name and confirmation number when asked. End after the agent confirms the +new appointment or gives a clear next step because it cannot be changed. +``` + +Do not tell the tester every sentence to say or how the agent must respond. A realistic tester should be able to clarify, react, and continue toward the goal. + +## Write focused success criteria + +Each criterion should answer one question that two reviewers could score the same way. + +| Criterion | Type | Role | Use it to check | +| --- | --- | --- | --- | +| `appointment_changed` | Yes/no (Boolean) | Required | The appointment was changed to the agreed date and time. | +| `no_false_confirmation` | Yes/no (Boolean) | Required | The agent confirmed success only after the scheduling result succeeded. | +| `final_details_clear` | Yes/no (Boolean) | Optional | The agent clearly summarized the final date and time. | + +Success criteria can return a Boolean, string, or number. Choose the simplest type that represents the result. Use **required** criteria only for conditions that must pass. Use **optional** criteria to observe quality without failing the whole simulation. Separate independent requirements so a failure shows what needs attention. + +Good criteria: + +- Describe an observable result. +- State exactly what counts as success. +- Allow different valid wording and conversational paths. +- Check both task completion and important guardrails. + +Avoid criteria such as "The call went well" or "The agent was helpful." If interaction quality matters, define it more clearly, such as "The agent acknowledged the caller's concern without blaming them and explained the next step." + +## Cover realistic paths + +Start with each core reason a customer calls. Add depth according to the frequency and cost of failure. + +| Path | Example | What it proves | +| --- | --- | --- | +| **Core outcome** | The requested appointment is available. | The main workflow succeeds. | +| **Natural variation** | The caller gives the date before their name. | The agent does not depend on one order of information. | +| **Missing or unclear information** | The caller says "next Friday" without a timezone. | The agent asks before acting. | +| **Recovery** | The scheduling tool reports an error. | The agent does not invent success and offers a next step. | +| **Guardrail** | The caller asks to bypass identity verification. | The agent protects the account even under pressure. | +| **Safe alternative** | The requested time is unavailable. | The agent offers valid choices without losing the goal. | + +Reuse a scenario across personalities when the outcome stays the same. Create +scenario variants with different variable values when the test data changes. +This creates useful variation without changing the business requirement. + +## Choose chat or voice deliberately + +| Mode | Best for | +| --- | --- | +| **Chat** | Fast iteration on prompts, tools, conversation logic, and success criteria. | +| **Voice** | Final validation of the complete audio experience, including transcription, speaking, and turn-taking behavior. | + +Build and debug a scenario in chat mode, then run important paths in voice mode before launch. Also use voice mode after changing the transcriber, voice, or turn-taking configuration. + + + A voice Simulation cannot represent every real caller, device, network, or environment. Pair automated tests with manual calls and regular review of production calls. + + +## Repeat critical tests and review the conversation + +Agent behavior can vary between runs. One pass shows that the agent **can** succeed; repeated passes provide stronger evidence that it succeeds consistently. Use more iterations for outcomes where a failure would harm a customer or the business. + +Read every failed transcript and sample passed transcripts. Ask: + +- Did the agent make a real mistake? +- Did the AI tester follow the scenario? +- Did the criterion reject a valid outcome? +- Was the run affected by a tool or configuration error? + +Keep the scenario and criteria stable when comparing agent changes. Update the test only when the expected customer outcome changes. + +## Avoid common pitfalls + +| Pitfall | Better approach | +| --- | --- | +| Prescribing every turn | Give the tester a goal, facts, constraints, and a stopping condition. | +| Passing because the agent said the task was complete | Check the strongest available evidence, such as the tool result and confirmed details. | +| Combining many requirements into one criterion | Use one criterion for each independent result or guardrail. | +| Testing only the happy path | Add ambiguity, failure, recovery, and refusal cases. | +| Treating one successful run as proof of reliability | Repeat business-critical simulations. | +| Editing a regression test until a change passes | Keep the original expectation unless the business requirement changed. | + +## Next steps + + + + Create and run a simulation suite. + + + Configure scenarios, personalities, and voice behavior. + + + Use variables, tool mocks, and reusable structured outputs. + + + Review results, repeat critical tests, and keep coverage current. + + diff --git a/fern/test/test-suites.mdx b/fern/test/test-suites.mdx deleted file mode 100644 index 2787d214f..000000000 --- a/fern/test/test-suites.mdx +++ /dev/null @@ -1,133 +0,0 @@ ---- -title: Test Suites -subtitle: End-to-end test automation for AI voice agents -slug: /test/test-suites ---- - - -Test Suites is being deprecated. It will be replaced by Simulations, a more powerful way to test your voice agents. You can keep using Test Suites in the meantime, and we'll share a migration guide once Simulations is ready. - - -## Overview - -**Test Suite** is an end-to-end feature that automates testing of your AI voice agents. Our platform simulates an AI tester that interacts with your voice agent by following a pre-defined script. After the interaction, the transcript is sent to a language model (LLM) along with your evaluation rubric. The LLM then determines if the interaction met the defined objectives. - -## Creating a Test Suite - -Begin by creating a **Test Suite** that organizes and executes multiple test cases. - - - ### Step 1: Create a New Test Suite - - Navigate to the **Test** tab in your dashboard and select **Test Suites**. - - Click the **Create Test Suite** button. - - ### Step 2: Define Test Suite Details - - Enter a title for your **Test Suite**. - - Select a phone number from your organization using the dropdown. - - Make sure the phone number has an assistant assigned to it (if not, navigate to Phone Numbers tab to complete that action). - - ### Step 3: Add Test Cases - - Once your **Test Suite** is created, you will see a table where you can add test cases. - - Click **Add Test** to add a new test case (up to 50 can be added). - - ### Step 4: Configure Each Test Case - - **Script:** Define how the testing agent should behave, including a detailed multi-step prompt to simulate how the customer should behave on the call. - - **Type:** Set the type of the test. 'Chat' simulates a text conversation, which we recommend because it is faster. 'Voice' simulates a call so you can hear a voice recording of the two assistants talking to each other. - - **Rubric:** List one or more questions that an LLM will use to evaluate if the interaction was successful. - - **Attempts:** Choose the number of times (up to 5) the test case should be executed each time the **Test Suite** is run. - - ### Step 5: Run and Review Tests - - Click **Run Tests** to execute all test cases one by one. - - While tests are running, you will see a loading state. - - Upon completion, a table displays the outcomes with check marks (success) or x-marks (failure). - - Click on a test row to view detailed results: a dropdown shows each attempt, the LLM's reasoning, the transcript of the call, the defined script, and the success rubric. - - -## Test Execution and Evaluation - -When you run a **Test Suite**, the following steps occur: - -- **Simulation:** An AI tester chats with or calls your voice agent, executing the pre-defined script. -- **Transcript Capture:** The entire conversation is transcribed, capturing both the caller's behavior and your voice agent's responses. -- **Automated Evaluation:** The transcript, along with your Success Criteria, is processed by an LLM to determine if the call was successful. -- **Results Display:** Each test case outcome is shown with details. Clicking on a test case reveals: - - The number of attempts made. - - The LLM's reasoning for each attempt. - - The complete transcript. - - The configured script and rubric. - -## Example Test Cases - -Below are three example test cases to illustrate how you can configure detailed simulation scripts and evaluation rubrics. - -### Example 1: Billing Support - -In this example, we will simulate a customer who is frustrated and calling about a billing discrepancy. - -**Script:** -```md title="Script" wordWrap -1. Express anger over an unexpected charge and the current bill appearing unusually high. -2. Try to get a detailed explanation, confirming whether an overcharge occurred, and understanding the steps for resolution. -3. End the call. -``` - -**Rubric:** -```md title="Rubric" wordWrap -The voice agent acknowledges the billing discrepancy respectfully without dismissing the concern. -``` - -### Example 2: Account Inquiry - -Unlike in the previous example, this time we will provide a more free-form script for the test agent to follow. - -**Script:** -```md title="Script" wordWrap -Simulate a customer inquiring about their account status with growing concern as unexplained charges appear in their statement. - -Your primary objective is to clarify several unexplained charges by requesting a detailed breakdown of your recent transactions and ensuring your account balance is accurate. - -Begin the call by stating your name and expressing concern over unexpected charges. Ask straightforward questions and press for more details if the explanation is not satisfactory. -``` - -**Rubric:** -```md title="Rubric" wordWrap -1. The voice agent clearly presents the current account balance. -2. The voice agent provides a detailed breakdown of recent transactions. -3. The response addresses the customer's concerns in a calm and informative manner. -``` - -### Example 3: Appointment Scheduling - -This time, we will spin up an even more detailed personality for the test agent. By showing these varied styles of scripts, we hope to show the flexibility of the **Test Suite** feature and how you can use it to meet your testing needs. - -**Script:** -```md title="Script" wordWrap -Simulate a customer trying to schedule an appointment with a hint of urgency due to previous delays. - -[Identity] -You are an organized customer who values efficiency and punctuality. - -[Personality] -While generally courteous and friendly, you are anxious due to previous delays in scheduling appointments, and your tone conveys urgency. - -[Goals] -Your goal is to secure an appointment at your preferred time, while remaining flexible enough to consider alternative timings if your desired slot is unavailable. - -[Interaction Style] -Begin the call by stating your need for an appointment, specifying a preferred date and time (e.g., next Monday at 3 PM). Request clear confirmation of your slot, and if unavailable, ask for suitable alternatives. -``` - -**Rubric:** -```md title="Rubric" wordWrap -1. The voice agent confirms the requested appointment time clearly and accurately. -2. The agent reiterates the appointment details to ensure clarity. -3. The scheduling process ends with a definitive confirmation message of the booked appointment. -``` - -### Frequently Asked Questions - - - - No, test calls cost you the same as regular calls. - - diff --git a/fern/test/voice-testing.mdx b/fern/test/voice-testing.mdx index 2c1644b60..4acdc3925 100644 --- a/fern/test/voice-testing.mdx +++ b/fern/test/voice-testing.mdx @@ -1,39 +1,98 @@ --- -title: Voice Testing -subtitle: Automated voice call testing for AI voice agents +title: Testing voice agents +subtitle: Learn to use Evals and Simulations to build reliable voice agents +description: Learn why voice agents need systematic testing, when to use Evals or Simulations, and how to combine them before launch. slug: /test/voice-testing --- ## Overview -Voice Test Suites enable you to test your AI voice agents through simulated phone conversations. Our platform connects two AI agents - your voice agent and our testing agent - on a real phone call, following your predefined scripts to evaluate performance under various scenarios. +Voice agents combine speech recognition, language models, voice generation, and tools. A small change to any part can improve one conversation and break another. Because the agent can respond differently to the same caller, one successful test call is not enough. -## How Voice Testing Works +Systematic testing turns your expectations into checks you can run again after a prompt, model, voice, transcriber, tool, or configuration changes. It helps your team agree on what the agent must do and find regressions before callers do. -1. **Simulation:** Our AI tester calls your voice agent and follows a script that simulates real customer behavior. -2. **Conversation:** Both AIs engage in a natural voice conversation, with the tester following your script guidelines. -3. **Recording:** The entire call is recorded and transcribed for evaluation. -4. **Assessment:** After the call, the transcript is evaluated against your rubric by a language model (LLM). + + Models and providers will change. The job your agent must do is more durable. Write tests around customer and business expectations, not the current implementation. + -## Benefits of Voice Testing +## Why testing pays off -- **Natural Interaction:** Test your voice agent in the most realistic scenario - actual phone calls. -- **Audio Quality Assessment:** Evaluate not just responses but also voice clarity, tone, and cadence. -- **End-to-End Verification:** Confirm that your entire voice pipeline works correctly from telephony to response. +Testing requires time up front, but its value grows as your agent changes. A useful test set helps you: -## Creating Voice Tests +- **Define success clearly.** Product, operations, and support teams agree on the expected behavior before launch. +- **Make changes with confidence.** Rerun the same checks after changing a prompt, tool, model, voice, or transcriber. +- **Separate regressions from normal variation.** Compare repeated results instead of relying on how one call felt. +- **Learn from real failures.** Turn support issues and production mistakes into tests that prevent the same problem from returning. +- **Evaluate upgrades faster.** Compare a new model or provider against the expectations you already defined. -You can create voice tests as part of a Test Suite: +## Use Evals for decisions and Simulations for outcomes -1. Navigate to the **Test** tab and select **Test Suites**. -2. Create a new Test Suite or edit an existing one. -3. When adding tests, select **Voice** as the test type. -4. Define your script and success criteria as detailed in the [Test Suites](./test-suites) documentation. +Evals and Simulations answer different questions. Most production voice agents need both. In a Simulation, an AI tester acts as the caller and adapts its responses while pursuing a goal. -## Voice Test Limitations +| Comparison | Evals | Simulations | +| --- | --- | --- | +| **Core question** | At this point in the conversation, did the agent make the right next decision? | By the end of the conversation, did the agent reach the right outcome? | +| **Scope** | Best kept focused on one or a few decision checkpoints, with a known conversation history | A complete conversation with an AI tester | +| **Conversation path** | Fixed context; checks what happens next | Flexible; the tester adapts as the conversation unfolds | +| **Best for** | Tool selection and arguments, routing, refusals, required questions, and specific responses | Task completion, recovery from problems, handoffs, caller behavior, and the overall experience | +| **Mode** | Text-based mock conversation | Chat or voice | -- Voice tests require more time to execute compared to chat tests. -- Each test consumes calling minutes from your account. -- Maximum call duration is limited to 15 minutes per test. +For example, use an **Eval** to check that an agent calls the rescheduling tool only after it verifies the caller. Use a **Simulation** to check that a caller can reschedule, hear the correct new time, and finish the conversation successfully. The Simulation should pass even if the agent asks the questions in a different valid order. -For detailed instructions on creating and managing test suites that include voice tests, see the [Test Suites](./test-suites) documentation. +Read the [Evals quickstart](/observability/evals-quickstart) for specific decision checks. Read the [Simulations overview](/observability/simulations-overview) for complete conversation testing. + +## Test in layers + +Plan coverage from the customer outcomes you need. When you run the tests, start +with narrow decision checks for fast feedback, then expand to complete chat and +voice conversations. Each layer catches a different kind of problem. + + + + Start with short tests for the choices that must be correct every time, such as verifying identity, choosing a tool, using the right arguments, refusing a restricted request, or selecting a handoff destination. + + + Run complete conversations without audio while you refine prompts, tools, recovery behavior, and success criteria. Chat Simulations run faster and cost less than voice Simulations. + + + Rerun important scenarios in voice mode to exercise transcription, pronunciation, pacing, and turn-taking. Use multiple iterations for critical scenarios because one pass does not prove consistent behavior. + + + Read failed transcripts and listen to a sample of recordings. Confirm that failures are fair and that passing conversations are genuinely good. Place controlled test calls when you need to verify phone-number routing, carrier behavior, voicemail, or other telephony setup. + + + + + Use chat for fast iteration, then use voice for the scenarios where speech and turn-taking can change the result. + + +## What good testing looks like + +A healthy test set is tied to real requirements and real caller behavior. It covers: + +- The main outcomes callers expect +- The critical decisions the agent makes along the way +- Both when an action should happen and when it should not +- Common failure and recovery paths +- High-risk privacy, compliance, financial, and handoff behavior +- Representative caller styles and important voice-specific cases +- Every important issue you have already fixed + +Do not aim to script every possible conversation. Prioritize the behavior with the greatest customer or business impact, then expand coverage as you learn from test runs and production calls. See [Plan test coverage](/test/plan-test-coverage) for a practical framework. + +## Next steps + + + + Decide which outcomes, decisions, risks, and caller behaviors to cover first. + + + Design complete conversations around durable customer and business results. + + + Protect the choices your agent makes at specific points in a conversation. + + + Set a testing cadence, investigate failures, and keep coverage current. + + From 627f6e47d48da1bcf6ff2f640a59b99c9be34b3d Mon Sep 17 00:00:00 2001 From: Scott Lowe Date: Fri, 4 Sep 2026 16:24:31 -0700 Subject: [PATCH 2/2] docs: clarify test variability --- fern/debugging.mdx | 2 +- fern/observability/simulations-overview.mdx | 8 ++++---- fern/test/run-and-maintain-tests.mdx | 10 ++++++++-- 3 files changed, 13 insertions(+), 7 deletions(-) diff --git a/fern/debugging.mdx b/fern/debugging.mdx index 2a88010d5..42dcfc28c 100644 --- a/fern/debugging.mdx +++ b/fern/debugging.mdx @@ -65,7 +65,7 @@ Start with these immediate checks before diving deeper: Test decisions at specific conversation points - + Test tools with sample data diff --git a/fern/observability/simulations-overview.mdx b/fern/observability/simulations-overview.mdx index 45f4e670c..2b1fa6e4b 100644 --- a/fern/observability/simulations-overview.mdx +++ b/fern/observability/simulations-overview.mdx @@ -72,7 +72,7 @@ Use [**Evals**](/observability/evals-quickstart) to test decisions at specific points in a conversation. Use **Simulations** to test outcomes across a complete conversation. Both are testing tools in Vapi, and they complement each other. -Evals verify specific, expected behavior. You script a mock conversation with user turns and expected responses, then check the result with exact match, regex, an AI judge, or tool-call checks. Evals are precise and repeatable, which makes them a good fit for unit-style checks. +Evals verify specific, expected behavior. You script a mock conversation with user turns and expected responses, then check the result with exact match, regex, an AI judge, or tool-call checks. Their fixed context makes them a good fit for focused, unit-style checks. Simulations test live interactions. An AI tester with a defined personality and intent improvises a conversation with your assistant or squad over voice or chat. Evaluations score the outcome with structured outputs, showing how the assistant or squad handles a realistic AI tester rather than a fixed script. @@ -80,12 +80,12 @@ Simulations test live interactions. An AI tester with a defined personality and | -- | -- | -- | | **What you're testing** | A decision at a specific conversation point | The outcome of a complete conversation | | **What you provide** | A scripted conversation with expected responses | An AI tester personality and an intent | -| **How it runs** | Repeatable, turn-by-turn checks | Dynamic; the AI tester improvises the conversation | +| **How it runs** | Fixed-context, turn-by-turn checks | Dynamic; the AI tester improvises the conversation | | **Transport** | Chat (mock conversations) | Voice, or text-only | | **Evaluation** | Exact match, regex, AI judge, tool-call checks | Structured outputs on the conversation outcome | -| **Reach for it when** | You need precise, repeatable checks | You need realistic end-to-end or voice behavior | +| **Reach for it when** | You need focused checks at a known point | You need realistic end-to-end or voice behavior | -Choose Evals when you want to lock down a specific response or verify a tool call's arguments with fast, repeatable checks. +Choose Evals when you want to lock down a specific response or verify a tool call's arguments with fast, rerunnable checks. Choose Simulations when you want to see how your assistant or squad performs against an AI tester with different personalities, multi-turn flows, squad handoffs, or voice. diff --git a/fern/test/run-and-maintain-tests.mdx b/fern/test/run-and-maintain-tests.mdx index 1283d8f51..4b0b9aca1 100644 --- a/fern/test/run-and-maintain-tests.mdx +++ b/fern/test/run-and-maintain-tests.mdx @@ -33,7 +33,7 @@ Run the smallest useful set while you work, then increase coverage as a change g Rerun affected tests whenever you change prompts, tools, models, voices, transcribers, call settings, or business rules. A result from an older assistant version doesn't validate the current version. -## Repeat Simulations +## Repeat critical tests AI conversations can take different paths from the same starting point. One passing Simulation shows that one path worked, not that the journey is reliable. @@ -41,7 +41,13 @@ AI conversations can take different paths from the same starting point. One pass - Repeat release checks based on the impact of failure and how much the result has varied before. - Use more iterations for critical journeys and intermittent failures when the extra evidence justifies the time and cost. -Compare the iterations instead of averaging them away. If one of five calls violates a critical requirement, treat the behavior as unstable until you understand why. See [reviewing and rerunning Simulation results](/observability/simulations-manage#review-run-results). +Compare the iterations instead of averaging them away. If any iteration violates a critical requirement, treat the behavior as unstable until you understand why. See [reviewing and rerunning Simulation results](/observability/simulations-manage#review-run-results). + +Evals use a fixed conversation context, but the assistant's response can still +vary between runs. An AI judge can vary too. Exact match and regex apply a +consistent rule to each response, but they don't make the response itself +consistent. Rerun critical or surprising Evals as separate runs before drawing +a conclusion. ## Review more than the pass or fail label