Estimating the Cost of an AI Project: Why Estimates Miss by Several Times

The cost of an AI project should not be estimated from the number of agents, tasks or people. It should be estimated from the number of rounds multiplied by the average context in each round. Estimates usually miss in three places: the extra rounds that retries and verification add, the tokens spent on thinking, and a cache that is not actually working. Between August and October 2026, the development team used AI agents for software work and data processing, and measured three sets of usage. Estimated from agent count alone, two jobs cost about 10 times and 5 to 8 times the estimate, and in the third the agent count grew from about 6 to 136.

The method: rounds times average context

An AI agent re-reads the context accumulated so far on every round. The cost of one agent is therefore roughly the number of rounds, times the average context length per round, times the unit price of cached reads. The number of agents only says how many workers there are. It does not say how many rounds each one runs or how much content each round re-reads, so estimates built on agent or task counts are commonly off by five to ten times. On AI adoption and development work, the development team uses this calculation to find the order of magnitude before quoting. If the estimate passes a threshold set in advance, the client hears first, and usage is measured while the work is under way.

The three sets of usage are laid out below. Costs are converted at list prices, and the work ran on the development team's own plan, so these are not actual invoices. The table shows only multiples and usage.

The third was a code audit. The audit workflow produced about 120 findings; assigning one verification agent to each makes the number of agents track the number of findings. After the change to one agent verifying 15 to 20 findings, the count fell to about 7 agents and the cost to about 5% of the original.

The development team works to four rules:

• Before any dynamic fan-out, compute the ceiling of findings times verifiers; if no number can be stated, do not write the workflow that way.
• Split large tasks into small ones and cap the rounds of each.
• Measure usage while the work runs, not after it ends.
• When the estimate passes a threshold set in advance, tell the client first.

Thinking spends tokens before any text appears

Letting a model think before it writes makes the output follow the rules more closely, but the thinking is counted in the output tokens as well. On the test site of an AI report-generation feature, one report written in a single pass with thinking enabled, even with effort set to low, produced 12,000 to 17,000 output tokens, of which only about 3,000 were the text itself. The rest was thinking. Writing one report took 15 to 75 seconds, and 2 of 13 test questions ran past the writing deadline and failed.

The development team did not turn thinking off. It split the same data package among several writers who work at the same time: one for each sub-question, one for the direct answer and heading, one for reminders, one for planning, and one for everything else. Each writer receives only the instructions for its own field, so total time is about that of the slowest writer. If one section fails, only that part stays empty and the program fills it from the data; the whole report fails only when every section fails. In the test, all 13 questions finished, and the slowest took 46 seconds.

Splitting trades more calls for shorter waiting, so the cost has to be re-estimated with the rounds-times-context calculation and cannot reuse the single-pass estimate. A trade-off like this also cannot be judged on one measure. Research on evaluating language models argues for looking at several metrics together rather than accuracy alone, because only then do the trade-offs show (Liang et al., 2023).

The cache threshold fails silently

Prompt caching makes repeated content cheaper, but it has a minimum length. The official documentation states that prompt caching has a minimum cacheable token count that depends on the model, and a prompt below that threshold is not cached even when it is marked with cache_control (Anthropic, Prompt caching).

For cost reasons, the production site of Cooking Monsters moved from gpt-4o to Claude Haiku 4.5 and added caching to the system prompt. Measured on that site: for the same workload, cost after the switch was about half of the original, and a cache hit cost about 27% less than no cache.

The risk is the length of the prompt. This system prompt ran about 4,900 tokens, and the model's minimum cacheable length is 4,096 tokens, so the margin was only about 800. That figure follows the official documentation at the time and may change. If someone later trims the prompt below the threshold, the cache stops working without an error. Cache reads simply fall to 0, the bill goes up, and nothing announces it.

The countermeasure is to make the failure visible. The log table records four kinds of tokens: input, output, cache write and cache read. Rates sit in an admin setting that can be changed, and a usage and cost dashboard sits on top, so when cost rises it shows which column moved. An estimate should also settle how long the prompt is, which model it runs on, what the threshold is, and who watches the cache-read volume. For keeping watch on usage and quality after launch, see AI quality and operations after launch.

In large batches, review costs more than production

Reordering paragraphs and tagging a batch of over a thousand documents is typical large-batch work. In two pilot rounds covering 62 documents, nearly all the spend went to review: the review cost over the two rounds was about 20 times the cost of tagging. The money went to checking whether the results were right, not to producing the results.

The full run therefore stopped sending every document to the most expensive model. A program first assembles a compact review sheet, so reviewers do not dig through the database or the original files. Haiku does the tagging, one subtask per document. Long documents are split first and merged afterward, and a result is accepted only when the validator exits with code 0, with no reliance on eyeballing. Sonnet reviews a sample of 10 to 15% chosen by risk, with long documents, scanned files, layout doubts and new sources first. Escalation conditions are written as rules: only flagged hard cases, plus anything the validator rejects, go to Opus.

The whole batch cost on the order of several hundred dollars, under a dollar per document. These are orders of magnitude converted at list prices, not an invoice.

The review side also needs sampling and program checks rather than a model grading itself. Known biases of model judges include a preference for certain answer positions, longer answers, or answers the model produced itself (Zheng et al., 2023). How to set the quality rules is covered in quality gates for batch content.

Budgets should be split by purpose

The quota for user-facing features should not be consumed by internal batch jobs or tests. A common mistake is using a key meant only for parsing end-user queries to run batch tagging: if the prompt is wrong, the whole batch is discarded and every call is pure waste, while legitimate front-end queries may fail for lack of quota. A batch pilot that runs on the backend key can likewise make the user-facing features on a test site fail completely for lack of quota.

The rule is to draw a hard line. Batch work, cleanup and tests never touch the key behind user-facing features. A batch pilot that does need that key reports its estimated cost first and runs only after approval. Bulk data processing uses a separate key or runs locally, and the periodic audit does not draw on user quota. On another project, the key is spent only on requests submitted through the admin interface, and when the key breaks the whole run stops instead of retrying a dozen times.

Splitting also makes the usage chart readable. When quotas are mixed and a runaway fan-out makes usage spike, the first reaction is often to suspect a leaked key, because the chart alone cannot tell normal work from a stolen key. Once purposes are separated, an anomaly is an anomaly.

Pilot small, then scale

Before a full run, run a small sample once, measure the score and the cost per item, then scale. For the document batch above, the full-run method was settled only after a pilot of 62 documents over two rounds, and the rules were then fixed so the next batch follows the same path. For evaluation before launch, see AI evaluation before launch.

1. Take a small, representative sample that includes the hard cases, such as long documents and scanned files.
2. Measure the quality score and the cost per item together; both need numbers.
3. Scale the pilot's usage to the full run with rounds times average context, and leave room for retries and review.
4. Set a cost threshold: if the estimate passes it, tell the client first, then keep measuring usage during the full run.

A pilot also makes “not doing it” a decision that can be taken before the big money is spent: if two pilot rounds show the cost is too high, the work can be stopped before the full run.

Which numbers should never be left to a model is covered in online proposals for wedding catering: prices always follow the specification.

When this method does not fit

• A feature with a single call and a fixed prompt: rounds and context are both fixed, so calls times tokens per call is enough and this method adds nothing.
• No sample to pilot and no checkable quality rules yet: rounds cannot be estimated, so collect samples and set rules before discussing a budget.
• A budget that must be fixed as one number before any pilot and cannot change afterward: this method produces a range that narrows only with measured values from the pilot.

Next step: what to prepare

Before talking, prepare four things: the list of tasks to hand to AI and the expected volume, a small representative set of sample data, checkable rules for what counts as passing quality, and the quota that user-facing features and internal batches will each use. Once those are ready, contact the development team and start by measuring a small pilot.