Quality Gates for AI Batch Content: Measure Existing Pages, Then Write Checkable Rules

Batch content written by AI tends to fail in two ways. The first is style drift: across a set of pages, section headings, image placement and caption length each drift in a different direction, and a reader can tell the pages were not written by one person. The second is fabrication: details the source never gave, such as material, schedule and warranty terms, get filled in by the model and read as true.

The fabrication risk is documented. NIST's generative AI profile lists information integrity among the risks and notes that generated content can make fact hard to tell from opinion (NIST, 2024). On a company website, a spec line the model invented is read as a statement from the company.

A better prompt alone does not remove either problem. What works is a set of gates in the pipeline. Each gate has a clear pass condition, a program judges whatever it can judge, and only the rest goes to a model or a person. POLISH Engineering Team uses this approach in its AI batch content and data processing work. The sections below take the gates one at a time.

Measure Existing Content First, Then Write the Rules

The live site of CHYIHAUR already has 90 case articles, and every new case from here on is built by AI. To make a new article read like the old ones, the first step is not a style guide. It is a count of what those 90 articles actually look like. The rules come from the counts, not from design preference.

The counts above date from summer 2026. NIST's AI Risk Management Framework (AI RMF 1.0) has four functions, GOVERN, MAP, MEASURE and MANAGE, and MEASURE covers analyzing, assessing, comparing and monitoring AI risk with quantitative, qualitative or mixed methods (NIST, 2023). Counting before writing rules follows the same order.

Rules Must Be Checkable by a Program

A rule counts only when a program can answer pass or fail. Section headings are limited to three fixed ones, and a program compares strings. Images sit in fixed slots, and a program checks that each slot has one. Captions have a length range, and a program counts characters. A condition such as natural tone cannot be answered by a program, so it stays out of the must-pass rules and is left to the sample review.

Counting also exposes defects in the existing data, and the defects go straight into the rules. About nine in ten existing image blocks had an alt attribute identical to the title, averaging 5 characters. For a reader who cannot see the image, that carries no information. The new rule requires the writer to look at the image first and write a description of 20 to 40 characters that does not equal the title.

Rule values need calibration against real review results. The upper limit was set at 40 characters and later moved to 50: descriptions of 41 to 49 characters turned out to be good, and a hard cutoff at 40 would have pushed reviewers to judge identical quality two different ways. A rule should block bad content, not good content.

Passing every automated check does not make the content right. Text and images still have to be checked page by page after a batch, because the two can be mismatched, for example when the text describes a building the picture does not show. The top of each case page should also be checked with phone and desktop screenshots, since a title can sit on top of the signage lettering and the signage can be cropped on the phone layout. The API and structure checks give no signal for either error, so matching text to images and reviewing screenshots stay with a person.

Rules also need upkeep. Sculley et al. observe that the fast progress of machine learning systems looks free but accumulates long-term maintenance cost, and the abstract lists data dependencies as one source of risk (Sculley et al., 2015). The rules depend on the existing data; when the data changes, the rules must be recalibrated, and the move of the alt limit above is one example. The checklist of Breck et al. lists the tests and monitoring a machine learning system should have before and after launch (Breck et al., 2017). Gates should be built the same way: able to block before launch and visible after it.

Leave a Field Empty When No Source Exists, and Ask for the Missing Material

Fields in the brief may take only information that actually appears in the source documents. If it cannot be found, the field stays empty and is not padded with filler. Invented material, schedule and warranty terms become false statements once published on a website.

A field such as construction difficulties is the most valuable one, and generation quality drops visibly when it is blank. When a field is blank, the process reports what is missing and asks a colleague to supply it, rather than writing around the gap. Known facts must not be left empty: information about CHYIHAUR itself, such as the planning and execution unit of a project, is filled in even when the brief omits it. Material, schedule and warranty terms belong to the project's client, so without a source those fields stay empty.

When the source of batch content is retrieved documents, how RAG works and where it fits is the place to start. For how a pipeline is tested offline before launch, see AI evaluation before launch.

Translation: Strings That Programs Compare Must Not Be Translated

CHYIHAUR's case site went live in four languages in 2026, at a scale of 90 cases, 27 categories, 83 brands and 7 technical articles. In translation, the block structure of the content (count, sequence, and the type, order, image and url fields) must not change, and only text fields are translated.

1. Some Chinese strings double as program identifiers, such as statuses, material matching and values sent to the backend, and translating such strings breaks things. When a dropdown option has no value, its display text is the submitted value, so a value must be added before the label is translated.
2. A check for partner organizations written as a comparison of partner.title with the Chinese word for client breaks once the site has several languages: the condition never holds, and the year disappears from the phone layout.
3. Message keys in next-intl must not contain a dot; otherwise the whole message provider throws and every message on the site fails. The key sets of the four languages must be identical, and a script compares the sets after translation.
4. Trade terms, such as illuminated lettering, take one rendering across all four languages.

The shared cause is that the translation model cannot know which strings the code refers to, so a script compares the languages after translation.

Review Costs More Than Labeling

A set of more than a thousand PDF documents needed paragraph reorganization and labeling. Two pilot rounds covered about 60 documents. The average of five scoring criteria rose from about 3.7 to about 3.9, and most documents came out better than the old version. Nearly all the spending went to review: across the two rounds, review cost about 20 times as much as labeling.

The conclusion was that the full run would not have the most expensive model read every document. The change has four parts.

• A program first produces a compact review packet, so reviewers do not dig through the database and source files.
• A mid-tier model reviews a risk-weighted sample of 10 to 15%, starting with long documents, scanned files, layouts that look doubtful and documents from new sources.
• Escalation is a written rule. Only when the boundary or coverage score is 3 or below and the cause lies on the labeling side does a document go to the strongest model. Documents that fail the validator are escalated as well.
• On the labeling side, each document is one sub-task, and long documents are split first and merged afterward. A document is accepted only when the validator exits with code 0, not on a person's visual judgment.

Full-run results: about 1,500 documents and about 29,000 items, over 90% with scenario subheadings. About 160 documents (about 10%) were sampled for review, and the average score of about 4 met the target. About 1,600 sub-tasks went to the smallest model, about 50 to the mid-tier model, and fewer than 10 to the strongest. The whole batch cost in the hundreds of dollars, under one dollar per document on average, with amounts converted from list prices rather than taken from an invoice.

Sampling rests on the premise that labeling quality can be measured. Snow et al. collected non-expert annotations through a crowdsourcing platform; on five tasks the non-expert labels agreed closely with expert gold standards, and the authors proposed bias correction to improve quality (Snow et al., 2008). The subjects were people, not models, so what carries over is the method: keep reference answers, measure real quality on a sample, and decide from that whether to tighten review. Model judges, on the other hand, have known preferences, including a preference for answers in a particular position, for longer answers and for answers the judge produced itself (Zheng et al., 2023). For that reason, anything a program can verify, such as verbatim matching and exit codes, is not left to a model judge.

Attach the Original Quote to Every Figure, Then Have a Second Reviewer Verify

Eligibility terms on an official data page, such as grade level, grade-point thresholds and language test requirements, are the content a model most easily rewrites in passing. The handling is to attach the original quote to every structured value, and a program verifies that the quote is an exact substring of the source and that each number appears inside the quote. Where the source says "or," the data says or; where it says "and," the data says and. A condition the source does not state is not added.

Extraction runs in two stages. The first extractor pulls the terms group by group; a second, independent reviewer then rereads each row, with a prompt that says to assume the first extractor made mistakes and to check grade level, grade-point thresholds, whether any test is missing, old versus new rules, sub-scores, undergraduate versus graduate, and or versus and. Both stages must pass the checking program, and rows that still need a person are recorded in a flags field and compiled into one table for the client to confirm.

Updates redo only what changed. After the official data is fetched again, a diff tool lists additions, disappearances and changes. A row that disappears is only a candidate for removal; the client is asked first and nothing is removed automatically. Only entries whose eligibility terms changed go back to AI for structuring, and other changes, such as quotas and calendars, are applied by program. An entry that differs only by an English parenthetical still counts as changed, because the quote must appear verbatim in the source. The first pass covered about 290 units and about 330 sets of terms, and the next two passes about 100 and about 60 units.

Estimate Cost by Rounds Times Context, Not by Agent Count

Cost is easiest to underestimate when AI agents do a large volume of work. Cost depends on how many rounds each agent runs and how much content is reread each round, so counting only agents misses the largest part; with dynamic dispatch, the agent count also multiplies with the number of findings.

The estimate is rounds × average context × the cache-read unit price. Before dispatching agents dynamically, calculate the ceiling of findings × verifiers; if that number cannot be stated, the design should not be used. The other practices are to split work into small tasks, cap the number of rounds and measure usage midway. When the estimate exceeds a threshold, it goes to the client first and the run starts only after agreement.

Quota must also be split by purpose. A common failure is a key meant only for parsing end-user queries being used to run batch labeling: if the batch prompt is wrong, every call in the thrown-away batch is pure waste, and legitimate front-end queries can fail once the quota runs out. A test site’s quota is usually small, and a batch test alone can use up the quota that users need, leaving the Q&A feature failing entirely. The answer is to draw lines: batch jobs, cleanup and tests never touch the key that serves users; a batch trial on that key reports its estimated cost first and runs only after agreement; and whole-batch data processing runs on the team’s own plan.

There is one more reason for the split. When runaway parallel calls make usage spike, the first reaction is often to suspect that the key has leaked, because a usage chart cannot tell normal work from a stolen key. When internal batches use a separate key or run locally, an anomaly on the chart is an anomaly. The key is spent only on requests submitted through the admin console, and if it breaks, the whole round stops instead of retrying more than a dozen times.

Caching is another place to save money, and it has a threshold that fails silently. On the production site for Cooking Monsters, a cache_control marker was added to the system prompt, which ran about 4,920 tokens, only about 800 above 4,096, the minimum cacheable length of the smaller model in use. The vendor's documentation states that prompt caching has a model-dependent minimum number of cacheable tokens, and that a prompt below the threshold is not cached even when it carries a cache_control marker (Anthropic prompt caching documentation). If the prompt is later trimmed below the threshold, cache reads drop to zero while the API raises no error. The countermeasure was to log four kinds of tokens (input, output, cache write and cache read) and build a usage and cost dashboard so a failure becomes visible.

The same principle extends to images and quotes, as in a character system for a large campaign and online proposals for wedding catering.

When This Approach Does Not Fit

• The batch is small. Handling pieces by hand is faster, and the up-front work of counting, writing rules and building validators does not pay back.
• The content itself must carry a distinct point of view, such as commentary. Rules can ensure consistency and no fabrication but cannot ensure a viewpoint.
• The source material is thin and blank fields are not acceptable. When the model must fill every field, missing information can only be invented.

What to Prepare Before Starting

Three things need to be ready: a sample of the existing content and its volume, the languages to be processed, and who owns the final review. The sample is used to count and set the rules, the volume sets the sampling rate and budget, the languages decide whether identifier checks are needed, and the final reviewer decides who takes the hard cases. When those are ready, contact POLISH Engineering Team with the state of the existing content.