RAG for Enterprise Knowledge Q&A: A Verbatim Verification Workflow

The definition of RAG

RAG stands for Retrieval-Augmented Generation. It works in two steps: first find the content related to a question in the company's own data, then hand that content to a language model and require the model to answer from it.

The term comes from the original paper by Lewis et al. in 2020, where the data source was a vector index of Wikipedia. In general use today, RAG means the structure of looking up data first and generating the answer second. Nothing prescribes which technique the lookup step uses.

What leadership wants to know is how to keep the model from making things up, and the three letters RAG do not answer that. RAG only guarantees that the model has seen the data, not that it follows the data. A model that has read the data can still add a sentence, change a number, or turn "many" into "always."

What matters is tracing answers to the source, not the choice of database

In procurement, the question raised most often is which vector database to use. The order should be reversed: first confirm that every sentence the model produces can point back to a specific passage in the source. If it cannot, users have no way to verify the answer and the company has no way to stand behind it.

A team member examines a stack of paper with a magnifying glass

In a data Q&A service built by the POLISH team, the corpus was long PDF documents written by users, about 1,600 in total. The source files came to about 9GB and were organized into about 13,000 verbatim excerpts.

This project did not use a vector database.

It did no embedding either. The service relies on a layer of verification that runs after retrieval.

Verbatim verification: the model picks excerpts, code compares, mismatches are dropped

The team split the work into two halves:

The model reads the whole document and picks continuous excerpts without changing a single character.
Code runs NFKC normalization on the source and on each excerpt, strips all whitespace, and locates the excerpt in the source with a string search.
Anything not found is dropped. So is anything shorter than 10 characters.

One real run: about 1,600 documents, about 15,000 excerpts output by the model, about 13,000 passed, and the dropped ones came to about a tenth. No document failed entirely. These are approximate figures from summing the output logs. Without this comparison step, the dropped excerpts would have gone into answers unchanged.

Fig. 1About a tenth of the excerpts the model picks fail the verbatim check

Long PDF documentsabout 1,600
Model picks excerptsabout 15,000
Code compares to the sourceNormalize, then locate by string search
Dropped on mismatchabout a tenthIncludes those under 10 characters
Passedabout 13,000

Rounded figures from one full run.

How the work is divided across the pipeline

The service produces a report based on the conditions the user selects. The flow is below. The model never touches the database and sees only the data package the code hands over.

Two people operate a conveyor belt that carries items along

Step	What happens	Performed by
Break down the question	Split the question into items to look up	Model
Fetch data	Pull a data package from the database and static files according to the conditions	Code
Compare	Compare the data against thresholds	Code, no AI call
Write	Write sentences from the data package	Model
Verify	Check sources and numbers; a failing sentence is removed whole	Backend verification layer

When something does not match

Numbers: all are placeholders, and code fills in the values from the data. Any number the model writes on its own, whether as digits or in Chinese numerals, causes the whole sentence to be removed.
Sources: every sentence must name the record it cites. If no source can be found, or the number does not match, the whole sentence is removed.
Original text: the model returns only the excerpt's ID, and the front end reads the source text live. A report stores only the ID, so once the data is taken down, the citation disappears with it.

Within the same batch of work, the team also built a customer-service Q&A version with stricter rules. A number in a key point must appear in the cited source or in the user's question. If verification fails, the system retries once with the reason attached; if it still fails, it drops only the problem points. If every point is dropped, it does not force an answer. It replies "not ready yet" and hands off to a person.

Retrieval in this version is BM25 keyword scoring plus synonym expansion. The confidence threshold is measured from test questions: it sits between the lowest score among questions the system can answer and the highest score among questions it should not answer. If the score falls short, the system lists the few closest questions and hands off to a person.

Fig. 2A low score or a fully stripped answer goes to a person instead of a forced reply

Keyword scoringBM25 plus synonym expansion
Compare to thresholdMeasured from test questions
Check each pointNumbers must be in source or question

Score falls short
List closest questionsHand off to a person
Check fails
Retry once with the reasonDrop failing points if it still fails
All points dropped
Reply “not ready yet”Hand off to a person

The flow in the customer-service version.

When vector search is needed

The weakness of keyword matching is that nothing is found when the user's wording differs from the source text. A synonym table covers part of the gap, but someone has to maintain it. Vector search is worth evaluating only in situations like these:

The way users phrase questions differs greatly from the wording in the source, and the synonym table cannot keep up.
The corpus is large and written inconsistently, so it cannot be organized into question-and-answer pairs in advance.

Even after switching to vector search, only the lookup step changes. Whether an excerpt the model picks traces back to the source is still decided by code comparison, regardless of the retrieval technique.

Prices are not left to the model

The AI proposal generation for Cooking Monsters follows the same principle: the model is not responsible for facts.

Pricing method, price per person, and add-on services are all set by specs in the admin panel. The backend overwrites the model's output with those specs and trims the number of dishes if it exceeds the spec limit. The model handles the parts that call for creative writing, such as the proposal name.

The data given to the model is narrowed first. Candidate dishes are pre-filtered with SQL: the season is inferred from the date, indoor or outdoor from the venue, and whether a dish is vegetarian from its description. Past cases with a similar service, price range, and headcount also go into the prompt. Prompts are managed in sections, every change is versioned, and any version can be rolled back.

Access control and personal data in data Q&A

A wrong answer is one kind of risk; an answer that exposes content not meant to be public is another. What the service above does:

The corpus is non-public by default and released only with an explicit status. The gate is enforced in the backend, not just hidden on screen.
Detail views return only masked identifiers. In output, only the matched string is masked for emails, identifiers, and phone numbers; the rest follows the source text.
Photos in PDFs that contain recognizable faces are all hidden first. Cross-validation found no face that slipped through and no faceless photo wrongly excluded.

Screening for personal data takes a dedicated face detector; classifying photo content is no substitute. Nearly 300 photos labeled "group photo of people" in that classification were actually empty hallways. The face detection is done by AI, and the team judged it reliable after spot checks.

For the security checks before a data Q&A service goes live, see the difference between vulnerability scanning and penetration testing.

Cost and time per document

The measurement conditions were: October 6, 2026, model claude-haiku-4-5, test environment, a real key, 12 documents. The average was USD 0.0166 per document, including the second model call made during verification; 12 documents came to USD 0.23 in total; each document took 12 to 32 seconds. These are test-environment measurements.

Trade-offs in the implementation

When code comparison falls short, leave the field empty

When assigning data to a category, code makes a first-pass comparison. First-pass comparison inevitably gets some wrong, and review has to correct those; another batch cannot be decided no matter how it is read. The field is left empty for that batch rather than forcing a value. An empty field only means one field is missing, while wrongly filled data carries its error out with every citation.

The model rewrites "many" as "always"

When summarizing key points, the model writes the source's "many" and "in principle" as "cannot" and "always." Once the tone shifts, the meaning shifts. The team added a rule requiring the source's tone to be preserved. When a user claims the source words it differently, the answer does not flatly deny the claim and instead expresses uncertainty.

Logs prove a call succeeded, not that the answer is right

A success in the logs only means the model responded and the format was correct; the logs do not show whether the answer is right. The only way to confirm it is to check the answer against the source text or the spec.

Precomputed answers versus real-time answers

Precomputed answers can be reviewed question by question beforehand, but a question phrased differently falls through. Real-time answers handle any phrasing, at the cost of verifying every sentence on the spot.

When RAG is not a fit

The data has not been organized. Documents are scattered across individual computers, versions differ, and no one can say which is the latest. Verification can only check against the source text the company provides, and when sources contradict each other, the answers will repeat the contradiction. Organize the data first, then talk about RAG.
Only general Q&A is needed. When questions do not depend on the company's own data, such as writing emails, translating, or asking common knowledge, RAG is unnecessary and a general-purpose model is enough.
A wrong answer is costly and no one is available to review. Verbatim verification stops the model from fabricating, but it cannot stop the source text itself from being out of date.

Preparing before an evaluation

If a set of documents already exists, start by picking a few of the questions users ask most, and attach to each the source passage judged to be correct. Only with this pairing can the approach to retrieval and verification be discussed. Contact the team directly, or read about AI services first. For choosing a vendor, see evaluating a software outsourcing vendor.