Evaluating an AI Q&A System Before Launch: Fixed Test Sets, Confidence Thresholds, and Abstention

A Q&A system that answers accurately in a demo is not necessarily reliable in production. Demo questions are chosen by people; production questions are unknown in advance and often fall outside the data. A Q&A system retrieves documents first and then has a model write the answer, an approach known as retrieval-augmented generation (Lewis et al., 2020); the principles and fit are covered in how RAG works. This article covers the next step: how to evaluate before launch, so problems are found before users run into any.

A citation does not prove an answer is grounded. One study evaluated several commercial generative search engines and found that responses often contain sentences the attached citations do not support, and that a considerable share of citations do not support the very sentence to which each is attached (Liu et al., 2023, based on human evaluation). Another study introduced the ALCE benchmark, whose automatic metrics for fluency, correctness, and citation quality correlate strongly with human judgment; on the ELI5 dataset, only about half of the best model's citations fully supported its claims (Gao et al., 2023). Both point the same way: evaluation should check whether each individual claim is supported by its source, not whether a whole answer reads smoothly.

Fixed Test Sets: Decide What Gets Tested First

A fixed test set is the foundation of the evaluation. The client supplies real questions that users are likely to ask, each paired with the source passage judged correct, plus a batch of questions the system should not answer, such as questions outside the data or questions that need case-by-case judgment. Once the set is fixed, every change to code, retrieval weights, or knowledge content is re-run against the same questions, so it is clear which ones improved and which ones regressed, instead of relying on impressions. Production-readiness checklists follow the same idea: the ML Test Score lists the tests and monitoring that should exist both before and after launch (Breck et al., 2017).

• Regression set: about 50 questions, each with a clear correct answer and scoring assertions, used to confirm that a change did not break questions the system already answered.
• Defense set: about 40 questions covering what should not be answered, out-of-scope questions, and deliberate attempts to mislead, with a record of acceptable handling such as abstaining or handing off to a person.
• Feel test: fewer than ten questions in the voice of real users, unscored; a person reads the output text directly, because a score cannot show whether an answer reads well.
• Pure-function checks: separate checks for ranking, defense, and literal wording rules, with no network and no keys; a run passes only with exit code 0.

Security work splits checks into repeatable vulnerability scans and scenarios designed by people for penetration tests. Q&A evaluation can borrow the same split: the fixed test set handles repeatable checks with comparable results, and hand-designed tricky questions fill the gaps the test set does not cover, then get added to it once found. The difference between the two kinds of check is covered in vulnerability scans versus penetration tests.

Replay: Swap Model Calls for Files

The cost problem in evaluation is that every re-run calls a real model. Batch tests consume the client's API quota, and outputs vary slightly from run to run, so results before and after a change cannot be compared question by question. The fix is to replace each model call with a file: a call without an answer is written as a request file, a stand-in answerer replies question by question, and the reply is written as a matching answer file. One round moves every question forward to its first unanswered call, in order: breaking down the question, writing the answer, and judging whether it passes; once answers are filled in, the next round runs. The same request always maps to the same file name, so a re-run never re-answers; after a code change, replay mode reuses the existing answers and the before-and-after differences can be compared question by question.

The trade-off should be stated plainly. The stand-in answerer is a small model, not the production model, so this offline evaluation only verifies that the pipeline, the validator, and the rules fit together; it cannot represent the production model's wording. A separate end-to-end test therefore stays in place, running the full flow with a real model in a test environment at about $0.02 per output. The offline mode has safeguards of its own, shown in the diagram under what stays inside the boundary.

Confidence Thresholds: Measured from Test Questions, Not Guessed

A Q&A system first has to decide whether it is confident enough to answer. The retrieval stage scores each candidate answer using BM25, a ranking method that comes from the probabilistic relevance framework (Robertson and Zaragoza, 2009); Chinese text is split into adjacent two-character units, synonyms are expanded, and the user-phrasing, keyword, and answer fields are weighted separately. There are two thresholds: the retrieval score, and a fit value (the retrieval score multiplied by coverage). Neither is set by intuition; both are measured from the test set.

The method splits the test set into two groups: questions the system should answer, and questions it should not. Take the lowest score among the first group and the highest among the second; the threshold sits between those two scores. If the two groups overlap, retrieval itself is not good enough, so retrieval or knowledge content must be fixed first, because raising the threshold only hides the problem. Any change to weights or knowledge content means measuring the threshold again.

Abstention is a classic problem in machine learning. Selective classification lets the user set an acceptable risk level, and the classifier rejects some inputs at test time so that risk stays within that level with high probability (Geifman and El-Yaniv, 2017). SimpleQA, a benchmark of short factual questions, sorts responses into three categories, correct, incorrect, and not attempted, and notes that an ideal model should decline to answer when uncertain (Wei et al., 2024). The approach here matches the principle of both: measure the acceptable risk first, then have the system withhold an answer when confidence falls short.

The Answer Gate: Source Every Claim, Drop What Fails Verification

A high retrieval score only means relevant material was found; it does not mean the model's answer is faithful to it. Ragas lists faithfulness as one dimension of RAG evaluation, meaning whether the model faithfully uses the retrieved content (Es et al., 2023); FActScore splits long text into small factual units, checks each one for support from a reliable knowledge source, and computes the share that is supported (Min et al., 2023). The answer gate in production works at a similar granularity, checking one claim at a time.

• The model answers only from the highest-ranked retrieved sources and may not cite anything beyond those sources.
• Every claim carries a source number, so a reader can go back to the original text and check.
• Any number in the answer must appear in the cited source or in the user's question.
• Evaluative wording cannot be added by the model, and the system makes no judgment about the user personally, such as asserting that the person qualifies or has good odds.
• The source's tone and scope are preserved: where the source says “many” or “in principle,” the answer cannot rewrite that as “cannot” or “always”; when a user says the source is worded differently, the system does not deny it and answers “not sure” instead.
• When verification fails, the system retries once with the failure reason; on the second pass it removes only the problematic claims.

The validator itself can be wrong, and only end-to-end tests expose it. A rule against passing judgment on the user that bans “falls short,” “below,” and “above” will block valid answers when the official rules are written exactly that way; the better rule drops those words and blocks only assertions about the user personally. Rules on title length and restatement-style answers work the same way: an end-to-end test that shows a valid question being blocked is what tells the rule to loosen. A rule that is too tight blocks answers that should go out, so rules and test set are tuned together.

No Answer Yet: Abstention and Handoff to a Person

The most important sentence in a Q&A system is “there is no prepared official answer for this question yet.” When confidence is short, the system says exactly that, lists the three closest existing questions for reference, logs a miss, and hands off to a person. The answer gate follows the same rule: when every claim has been removed, the system does not force an answer and routes to a person instead; when verification fails on two attempts, it returns an error and does not charge the usage quota. Logged misses add up to a list of what knowledge content to add in the next round. Abstention is not a failure: it passes uncertain questions to a person and keeps what the system does say within what can be verified.

Structured Data: Quote Every Value, Review with a Second Pass

A Q&A knowledge base often starts by turning official documents into structured conditions, such as eligibility, levels, and qualifying thresholds. Numbers in official conditions are the easiest thing for an AI to rewrite quietly, so every structured value carries a snippet of the original text: code verifies that the snippet is an exact substring of the source and that each number sits inside the snippet, “or” and “and” follow the source, and anything the source does not say is not added. A second, independent reviewer then rereads, with a prompt that explicitly tells it to assume the first pass made mistakes and to go through every row again. Both passes must clear the code checks, and rows that still need a human are flagged and handed to the client in one table.

After the official data is updated, a diff lists new, removed, and changed items; only items whose conditions changed go back to the AI, and code applies the rest. Removed items are only candidates: the client is asked first, and nothing is taken down automatically. Even a change that differs only by the English text in parentheses is redone, because the snippet must appear verbatim in the source and may no longer match once the English is removed. The first pass covered nearly 300 official data pages and extracted several hundred sets of conditions; the two later incremental runs covered about 100 and about 60 pages. Cost is estimated before each run, and larger runs are quoted first.

After Launch: Failures That Look Like Success

The hardest failure in AI automation to catch is not a crash but one that looks like success. An automated run can finish with most of its outputs invalid and no error message at all, for example because the category list was never passed to the model. Type checking cannot catch it; only a wiring check that verifies the call is really connected can, and the assertion has to compare the real call, since a weak assertion lets the failure through. Failures of this kind take a few common forms:

• Writes that look successful: an update statement affected zero rows because its condition did not match, the function did not check the affected-row count, and the return value looked exactly like success, which left the data mislabeled. The countermeasure is to read the data back after writing and confirm the change took effect.
• Tasks that get missed: a newly added task never gets wired into the scheduler and can go unnoticed for days. The countermeasure is a wiring check that every task is actually called by the schedule.
• Connecting to the wrong database: connected to a different server, the pending list returned an empty array and the screen said nothing was pending, which looks exactly like the case where there really is nothing. The countermeasure is to confirm the production database first and to check “no data” and “wrong database” separately.
• Logs only prove a response: a call log showing success means only that the call succeeded, not that the answer was right. Correctness comes from the test set and source verification, and from spot checks that continue after launch.

After launch, a Q&A system produces usage events such as questions, abstentions, and misses, and the quality of that data has to be managed first for later analysis to be trustworthy. The development team's approach at the intake point is to maintain the event specification from a single source and validate on arrival; anything invalid is rejected, and only the reason and path are stored, never the content. Personal data is scanned again with the same patterns, and a hit clears the whole parameter instead of masking it. Once a day, rejection, loss, and duplication rates are aggregated, and a light turns on when a threshold is exceeded. A reconciliation that is not connected is labeled “not connected” in the interface, never shown as normal.

Slow Responses: Keep Thinking On, Split the Work in Parallel

When generation takes too long, the instinct is to turn off the model's thinking to gain speed, but the cost is rule compliance. In a test environment with a real model, writing the whole output in a single pass with thinking on spent more than 10,000 tokens thinking before writing, took about 15 to 75 seconds, and timed out on two of a little over ten questions; lowering the thinking effort made no difference. With thinking off, responses were much faster, but sentences removed by validation rose to about three times as many and most questions could not produce a title, so thinking stays on.

The fix is to split the work in parallel: the same data package goes to several writers at once, one per sub-question, one for the direct answer and title, and one each for reminders, planning, and the remainder, and each writer receives only the prompt for its own field, so total time is roughly that of the slowest writer. If one section fails, only that block is left empty and code fills it in from facts in the data; only when all sections fail does the whole output fail. After the split, the same set of questions all completed, and the slowest took under 50 seconds. The user side polls for the result and receives an acknowledgment right after submitting, so no connection sits waiting. Live Q&A has an overall deadline of about 45 seconds, below the 60-second default read timeout of a reverse proxy.

Division of Work in Evaluation

Evaluation is not one party's job. Questions have to represent real usage and correct answers have to be judged by people who know the material, and only the client can do both; scoring, measuring thresholds, and tuning retrieval belong to the development team; and results are reviewed jointly, because questions that are refused or fail validation sometimes point to gaps in knowledge content rather than errors in the system. The NIST AI Risk Management Framework organizes risk management into four functions, GOVERN, MAP, MEASURE, and MANAGE, and MEASURE uses quantitative, qualitative, or mixed methods to analyze, assess, benchmark, and monitor AI risk (NIST, 2023); a fixed test set and threshold measurement before launch can be seen as one concrete way to carry out MEASURE.

For how this kind of Q&A and evaluation work is done, see AI services; for turning a multi-round quote request into a four-step graded menu, see the Cooking Monsters case.

When It Does Not Fit

• Internal trials only, where no one makes decisions based on the answers: the cost of a fixed test set and thresholds outweighs the benefit, and launching to observe is enough.
• No one can label the correct answers: without correct answers, the threshold cannot be measured, so find someone who can make that call first.
• Source documents contradict each other: sort the data out before evaluating, because otherwise the evaluation measures the contradictions in the data, not the performance of the system.

Next Step: Prepare the Test Set

1. 20 of the questions users ask most often, each with the source passage considered correct.
2. A few questions the system should not answer, such as questions outside the data or questions that need case-by-case judgment.

Once these are ready, talk with the development team about the test set and evaluation method, measure a first threshold with these questions, and then decide which knowledge content to add.