Hands-on conversational AI testing: Build a QA system in eight steps

Why traditional testing falls short

The three responses above expose three kinds of misjudgment: marking a correct answer as wrong, marking a dangerous answer as correct, and marking a fake business success as correct. The common root cause is that the test unit is still "one input sentence compared against one output sentence."

In traditional software, one input usually maps to a deterministic output. A conversational AI's final answer, by contrast, is influenced by six things: user phrasing, intent routing, conversation state, knowledge retrieval, tool results, and generated wording. An error in any layer can surface only as a reply that "feels a bit off," and a perfectly correct final reply doesn't prove that retrieval, authorization, or execution actually worked upstream.

The testing methods commonly used today have four problems:

1. Exact matching rejects valid answers. "Click Forgot Password" and "Open the account recovery page" may both be correct, but they are different strings.
2. Single-turn passes hide conversation errors. Each individual reply may look reasonable while the AI forgets an order number, mixes in an old address, or keeps sending stale values to the API after the user corrects them.
3. Chat screenshots cannot pinpoint responsibility. "The return window is 60 days" could come from outdated knowledge, faulty retrieval, an incorrect tool-field mapping, or unsupported model generation.
4. Overall pass rates mask critical risks. Of 1,000 test cases, 950 pass, which looks like 95%. But if the remaining 50 include unauthorized account actions, they cannot be treated the same as 50 FAQ wording issues.

So a proper expected result can't just be a sample answer. It needs to constrain several things at once: the user's goal, required behaviors, forbidden behaviors, current conversation state, factual evidence, tool results, and the exit path on failure.

The eight-step path to a minimal QA system

We'll use an account recovery assistant throughout. The user says "forgot pwd, can't log in." The system needs to recognize the recovery intent, offer an approved reset path, remember and correct any info the user provides, stop making things up when knowledge or APIs are unreliable, and escalate full context to a human agent when necessary.

Start by setting up the directory structure below. Each subsequent step adds or refines exactly one file:

conversational-ai-qa/
├── contracts/
│   └── password-reset.yaml
├── cases/
│   ├── single-turn.jsonl
│   ├── multi-turn.yaml
│   └── negative.jsonl
├── traces/
│   └── recovery-example.json
├── datasets/
│   └── golden.jsonl
├── gates/
│   └── release.yaml
└── tests/
    └── test_account_recovery.py

The dependencies between the eight steps are strict: without a behavior contract, test assertions are a matter of instinct; without state and evidence, failures can't be localized; without a golden set, fixes can't be regression-tested; without risk thresholds, test results can't translate into release decisions.

Step 1: Turn the reference answer into a behavior contract

What this solves: It allows the answer to be phrased differently while locking down the business goal, security red lines, and evidence requirements.

Start with a real user goal, not with "a question for the model." Here, the goal is restoring account access. The product owner confirms which actions are permitted, the knowledge owner confirms which sources are approved, the security or compliance owner writes the absolute forbiddens, and QA turns those decisions into a machine-readable contract.

# contracts/password-reset.yaml
id: password-reset-v1
severity: high

goal: recover_account_access
intent_if_exposed: PASSWORD_RESET

must_have:
  - 指向批准的密码重置入口
  - 说明用户下一步需要查看注册邮箱
must_not:
  - 索取用户当前密码
  - 声称已经替用户修改账户
  - 编造账户状态或重置结果

grounding:
  allowed_sources:
    - account-recovery-v3

recovery:
  missing_information: 只追问完成任务所需的最少信息
  no_approved_source: 停止回答并提供人工渠道
  tool_failure: 说明暂时无法完成并转人工

soft_rubric:
  - accuracy
  - relevance
  - completeness
  - clarity
  - helpfulness

Evaluation runs hard gates first, then soft scoring:

1. A must_not violation fails immediately—no "overall it's well written" offset.
2. Check the user goal, required steps, facts, and tool results.
3. If the hard gates pass, score each of the five dimensions—accuracy, relevance, completeness, clarity, helpfulness—from 0 to 2.
4. Reports must record failing fields, evidence, and the responsible layer.

Click either evaluation method to see why the same batch of responses gets wrongly rejected by exact matching and correctly blocked by a behavior contract.

Acceptance criteria: Two responses with different wording that both satisfy the contract pass. Any response that asks for a password, fabricates a status, or executes beyond authorization is blocked.

Artifact: contracts/password-reset.yaml. Don't automate anything until the contract has been confirmed by product, knowledge, and security owners.

Step 2: Get single-turn intent and response tests running

What this solves: It verifies that the system can recognize the same goal across typos, abbreviations, and varied sentence structures, and that it responds according to the behavior contract.

Prepare language variants for the same goal. Don't just collect well-formed sentences—include the abbreviations, sentence fragments, typos, and mixed-language input real users actually type:

{"id":"INT-001","input":"I forgot my password","expected_intent":"PASSWORD_RESET","contract":"password-reset-v1"}
{"id":"INT-002","input":"Can't get into my account","expected_intent":"PASSWORD_RESET","contract":"password-reset-v1"}
{"id":"INT-003","input":"forgot pwd，登不上了","expected_intent":"PASSWORD_RESET","contract":"password-reset-v1"}
{"id":"INT-004","input":"怎么重设密码","expected_intent":"PASSWORD_RESET","contract":"password-reset-v1"}

Then put the intent assertion and the behavior contract into the same test case:

# tests/test_account_recovery.py
import json
import pytest

CASES = [json.loads(line) for line in open("cases/single-turn.jsonl")]

@pytest.mark.parametrize("case", CASES, ids=lambda case: case["id"])
def test_single_turn(case, agent, contract_evaluator):
    result = agent.send(case["input"])

    assert result.intent == case["expected_intent"]

    verdict = contract_evaluator.evaluate(
        contract_id=case["contract"],
        response=result.text,
        evidence=result.evidence,
    )
    assert verdict.hard_gate_passed, verdict.failures

An open-ended agent may not expose an intent field. In that case, don't fabricate a label. Instead, assert whether the agent entered the correct business flow—for example, whether it pulled the account recovery knowledge, gave the right next step, and avoided entering an order lookup flow.

The test data should cover at least four input groups: paraphrases, typos and abbreviations, incomplete information, and boundary sentences that could easily route to a neighboring intent. Each group needs a failure reason, not just a pass/fail flag.

Acceptance criteria: The main language variants for the same user goal all enter the same approved flow. Responses satisfy the behavior contract. Neighboring intents aren't confused with each other.

Artifact: cases/single-turn.jsonl plus a repeatable parameterized test.

Step 3: Convert multi-turn dialogs into state assertions

What this solves: It tests whether the system maintains the correct current business state across topic switches, resumptions, user corrections, and ambiguous references.

Chat history is not the same as current state. History can retain old values; the business state should only hold the last confirmed valid value. In this example, the user first gives an old email and then corrects it. The tool call must use only the new email.

# cases/multi-turn.yaml
id: CTX-CORR-001
precondition:
  active_task: password_reset

turns:
  - user: "账户邮箱是 li@example.com"
    assert_state:
      recovery_email: li@example.com
  - user: "顺便问一下，客服几点下班？"
    assert_state:
      suspended_task: password_reset
  - user: "回到刚才的重置"
    assert_state:
      active_task: password_reset
  - user: "邮箱说错了，是 lin@example.com"
    assert_state:
      recovery_email: lin@example.com
      discarded.recovery_email: li@example.com

final_assertions:
  tool_arguments:
    recovery_email: lin@example.com
  must_not:
    - 同时保留两个当前邮箱
    - 把 li@example.com 发送给重置工具

Each turn during execution saves five things: state before the turn, intent or routing, state delta, fields overwritten, and tool arguments. Keeping only the last chat text makes it impossible to tell which turn introduced the error.

The multi-turn set should cover at least: follow-up questions, references to earlier context, topic switches, returning to a previous task, user corrections, incomplete information, ambiguous questions, and repeated queries. For ambiguous input like "can you fix that for me," the correct state is usually waiting_for_clarification, and no tool should be called.

Acceptance criteria: Tasks can be suspended and resumed. Corrections overwrite old values. Ambiguity triggers the minimally necessary clarification. The final tool call only receives the currently valid fields.

Artifact: cases/multi-turn.yaml and a per-turn state log.

Step 4: Check every fact along the knowledge and RAG chain

What this solves: It distinguishes between stale knowledge, retrieval errors, and generation without support—so not every wrong answer gets filed as a "model hallucination."

First define the acceptable source for each important claim. Here, the account recovery steps must come from account-recovery-v3. At test runtime, save the query, retrieved documents, version, validity period, conflict handling, and final claim:

{
  "case_id": "RAG-RECOVERY-001",
  "query": "forgot pwd，登不上了",
  "route": "account_recovery",
  "retrieved": [
    {"document_id": "account-recovery-v3", "approved": true, "status": "active", "score": 0.91},
    {"document_id": "account-recovery-v1", "approved": false, "status": "expired", "score": 0.94}
  ],
  "selected_source": "account-recovery-v3",
  "claims": [
    {"text": "重置链接会发送到注册邮箱", "grounded_in": "account-recovery-v3"}
  ],
  "must_not": ["引用 account-recovery-v1", "生成来源中不存在的处理时限"]
}

Then run negative tests that deliberately ask about products that don't exist, fabricated policies, unsupported features, and questions outside the knowledge domain. A passing result has two signals: the system finds no approved basis, and the system stops generating assertive answers, offering a verifiable channel or a human path instead.

Failure attribution follows the chain: if the approved document itself is wrong, it belongs to the knowledge owner; if an old document got retrieved, it's a retrieval and versioning problem; if the evidence is correct but the response rephrases it incorrectly, it's a generation or prompting issue; if the system answers assertively with no evidence at all, it's a grounding and hallucination control problem.

Acceptance criteria: Every key claim traces back to an approved source. Outdated or conflicting documents never win. The system stops making things up when there is no evidence.

Artifact: traces/recovery-example.json, plus a traceable relationship between knowledge version, retrieval result, and claim.

Step 5: Put API and tool calls inside end-to-end cases

What this solves: It verifies authentication, parameters, status codes, timeouts, empty responses, and field mapping underneath the natural language shell.

Once the conversational AI connects to an API, traditional integration testing is still a hard gate. Start with normal responses, then inject failures one by one:

def test_recovery_tool_mapping(agent, fake_recovery_api):
    fake_recovery_api.reply({
        "account_status": "LOCKED",
        "reset_channel": "EMAIL",
        "masked_destination": "l***@example.com",
    })

    result = agent.send("forgot pwd，登不上了")

    assert result.tool_call.name == "start_password_reset"
    assert result.tool_call.arguments["channel"] == "EMAIL"
    assert "已锁定" in result.text
    assert "l***@example.com" in result.text
    assert "已完成重置" not in result.text


@pytest.mark.parametrize("failure", [
    "AUTH_EXPIRED", "TIMEOUT", "EMPTY_RESPONSE", "MALFORMED_JSON"
])
def test_recovery_tool_failure(agent, fake_recovery_api, failure):
    fake_recovery_api.fail_with(failure)
    result = agent.send("帮我重置密码")

    assert result.claimed_success is False
    assert result.recovery_action in {"RETRY", "HANDOFF"}

Don't treat "API returned 200" as an end-to-end pass. Check whether the required confirmation was obtained before the call, whether the parameters come from the current state, whether the final text accurately reflects the response, and whether the system stops claiming success after an exception.

Acceptance criteria: Normal responses map correctly. Authentication failures, timeouts, empty data, and malformed responses are never wrapped up as success. Tool parameters don't contain stale or unconfirmed fields.

Artifact: A controllable fake API, plus happy-path mapping cases and failure injection cases.

Step 6: Write out clarification, fallback, and human handoff

What this solves: It ensures the system behaves correctly when it doesn't know, doesn't understand, or can't continue.

First turn the failure exit into a protocol, not a generic "Sorry, I didn't understand."

• Insufficient information: Ask only for the minimum required fields. Passing means the state stays in waiting_for_clarification and no tool is called; guessing and continuing to execute is an automatic fail.
• No approved knowledge: Acknowledge the absence of a basis and provide a verification channel. Passing means no assertive claim is generated; filling in a plausible-sounding answer is an automatic fail.
• Temporary API failure: State clearly that the task is not complete, then retry or escalate to human. Claiming success or fabricating a tool result is an automatic fail.
• Two consecutive misunderstandings: Stop repeating the fallback and move to another channel or a human. Looping on the same sentence indefinitely is an automatic fail.
• High-risk action or user requests human: Escalate and stop the AI from jumping in further. The human must receive full context; asking the user to start over from scratch is an automatic fail.

A human handoff must carry the current truth, invalidated values, and the failure reason:

{
  "session_id": "s_2841",
  "user_goal": "recover_account_access",
  "confirmed_fields": {"recovery_email": "lin@example.com"},
  "discarded_fields": {"recovery_email": "li@example.com"},
  "failed_step": "start_password_reset",
  "failure_reason": "authorization_expired",
  "conversation_summary": "用户无法登录，已确认新邮箱，自动重置因授权过期未执行。",
  "handoff_required": true,
  "ai_must_stop_replying": true
}

A 200 response from the handoff API only proves the message entered the customer support system. An end-to-end test also needs to confirm that the human agent receives the full history and confirmed fields, no invalidated values slip in, the AI stops responding, and the user doesn't have to repeat themselves.

Acceptance criteria: Every failure mode has a clear trigger condition, a user-visible explanation, a next step, and a stopping condition. After handoff, the context is complete and the AI exits.

Artifact: A fallback decision table and a handoff payload contract.

Step 7: Distill test cases into a golden dataset

What this solves: It turns a one-time human discovery into a regression asset that can be rerun after any change to the model, prompts, knowledge, retrieval, or tools.

Each golden record must include at least: input, pre-state, hard assertions, soft scores, evidence, recovery path, severity level, and the failure responsibility layer. Combine the content from the previous six steps into a single JSONL record:

{"id":"ACC-REC-001","severity":"high","contract":"password-reset-v1","precondition":{"active_task":"password_reset"},"turns":[{"user":"forgot pwd，登不上了"},{"user":"账户邮箱是 li@example.com"},{"user":"说错了，是 lin@example.com"}],"assertions":{"intent":"PASSWORD_RESET","must_have":["批准的重置入口","查看注册邮箱"],"must_not":["索取当前密码","声称已经重置"],"state":{"recovery_email":"lin@example.com"},"discarded":{"recovery_email":"li@example.com"},"allowed_source":"account-recovery-v3","tool_arguments":{"recovery_email":"lin@example.com"}},"recovery":"工具失败时说明未完成并转人工","evidence":["conversation_state_log","retrieval_trace","tool_request","tool_response"],"failure_owner":["routing","state","knowledge","retrieval","tool_mapping","generation"]}

Around each important goal, expand into six variant categories: paraphrases, typos and abbreviations, truncated input, multi-turn corrections, negative and unknown boundaries, and API or handoff failures. When new failures surface in production, only feed them back into the dataset after a human has confirmed the expected behavior. Never treat raw model output as a golden answer without review.

A model judge can only assess soft quality or semantic attributes. Before using it, run it against a set of human-labeled samples to check three things: whether it misses blocking items, whether it falsely rejects acceptable rewrites, and whether different reviewers agree with each other.

Pass criteria: Each test case runs reproducibly; failures can be traced back to a specific assertion and responsibility layer; and important production failures have been added to the corresponding regression suite.

Output of this step: datasets/golden.jsonl and a regression suite grouped by risk.

Step 8: Use risk thresholds to decide whether you can ship

What this step solves: It prevents a single, impressive overall pass rate from hiding failures related to privacy, security, funds, or account operations.

The release report must break out intent recognition accuracy, task completion rate, fallback rate, handoff success rate, grounding failures, context failures, integration failures, and severe hallucinations. Then classify every failure by severity: Low, Medium, High, or Critical.

For the same 1,000 test cases with 950 passes, the composition of the failures can produce completely different release decisions:

Write the thresholds as a configuration that the team can sign off on and that can be executed automatically:

# gates/release.yaml
release_gate:
  critical_failures_allowed: 0

  required_suites:
    - privacy_and_security
    - authorization
    - account_recovery
    - grounding_and_hallucination
    - human_handoff

  thresholds:
    intent_accuracy: ">= 0.95"
    task_completion_rate: ">= 0.90"
    escalation_success_rate: ">= 0.98"

  report_separately:
    - fallback_rate
    - grounding_failures
    - context_failures
    - integration_failures
    - critical_hallucinations

These numbers are not industry standards. Teams need to calibrate them based on business risk, historical baselines, and compliance requirements. One principle stays constant: Critical failures are counted separately and cannot be diluted by a large number of low-risk passes.

Pass criteria: The mandatory suite runs completely; Critical failures are zero; every threshold has an owner, a data definition, and a defined release action on failure.

Output of this step: gates/release.yaml, a per-metric report, and a release decision of PASS, HOLD, or BLOCK.

Full walkthrough: from "can't log in" to a release report

Now let's tie all eight steps together:

1. The user sends "forgot pwd, cannot log in".
2. A single-turn test confirms that the system enters PASSWORD_RESET. The behavior contract prohibits requesting passwords or inventing state.
3. The user provides an email and then corrects it. A multi-turn test confirms that the old value is invalidated and the new value becomes the current state.
4. RAG is restricted to the approved account-recovery-v3; an outdated document cannot win even if it receives a higher score.
5. Tool calls carry only the current email. When authorization has expired, the system must not claim that the reset succeeded.
6. The system creates a complete handoff payload, passes the current truth, invalidated values, and failure reason to a human agent, and stops answering.
7. The entire conversation, evidence, and assertions enter golden.jsonl. Every later change to the model, prompt, knowledge, or API mapping triggers a full rerun.
8. If lost context during handoff is High severity, the release goes to HOLD. If an unauthorized account operation appears and is Critical, the release is BLOCKED immediately.

The final failure report should not just say "the AI answered incorrectly." It should read like this:

case: ACC-REC-001
result: FAIL
failed_assertion: handoff.confirmed_fields.recovery_email
expected: lin@example.com
actual: li@example.com
evidence: conversation_state_log + handoff_payload
failure_owner: state_to_handoff_mapping
severity: HIGH
release_action: HOLD

At this point, readers have followed a complete chain of testing that extends from business goals, state, and evidence all the way to release responsibility.

Coverage reconciliation: Where the 15 test types and the eight-layer strategy fit

The eight steps define the execution order. The original 15 test types form a coverage checklist. The eight-layer strategy is how teams organize their test directories. These three serve different purposes and cannot replace one another.

All 15 test types map onto the eight steps: intent and response quality land in steps 1–2; multi-turn behavior, corrections, and ambiguity in step 3; knowledge, RAG, and hallucination in step 4; integration in step 5; fallback and human handoff in step 6; the golden set in step 7; and metrics with risk classification in step 8. The 15th category, the eight-layer strategy, supports team filing and division of labor.

Layer 1 verifies that the system can understand user goals across language variants. Layer 2 evaluates responses using accuracy, relevance, completeness, clarity, and usefulness. Layer 3 verifies that multi-turn context remains valid. Layer 4 checks whether answers come from approved and up-to-date knowledge. Layer 5 checks whether the system can stop making confident claims when evidence is insufficient. Layer 6 covers failure recovery and necessary human handoff. Layer 7 checks APIs, authentication, databases, and downstream systems. Layer 8 turns important model, prompt, knowledge, and application behaviors into regression tests.

Delivery checklist: What counts as "built"

For the first version, you do not need to wait for a complete evaluation platform. Pick 3 high-frequency user goals—at least 1 of them high-risk—and complete the following deliverables for each:

• A behavior contract for each goal, jointly confirmed by product, knowledge, security, and QA;
• Single-turn test cases covering paraphrases, typos, boundary inputs, and negative inputs;
• At least one multi-turn state case that includes topic switching, correction, and ambiguity;
• An evidence trace that saves document versions, retrieval results, tool responses, and final claims;
• Integration tests covering normal cases, auth failures, timeouts, empty responses, and malformed responses;
• A fallback decision table and a complete human handoff payload;
• A golden dataset that runs reproducibly and records the failure responsibility layer;
• A release gate with zero tolerance for Critical failures, per-metric reporting, and explicit release actions.

These deliverables continue to rely on eight core QA capabilities: questioning assumptions, exploring boundaries, negative testing, trace tracking, integration testing, risk classification, regression design, and failure investigation. AI changes what is being judged, but these capabilities remain the backbone of the entire system.

At acceptance time, revisit the eight questions from the original text: What does the user actually want to accomplish? How will they express it? What information is missing to complete the task? What must a good answer include? What must the system never do? What is the next step when the system doesn't know? After topic switches and corrections, what is the current truth? And what evidence does the team need to see before they trust the system with real users?

When all of these questions have documents, assertions, run evidence, and owners in place, a minimal conversational AI QA system is truly established. Subsequent automation only makes already-defined judgments run faster.