Cost per Correct Answer: How to Measure Which RAG Pipeline Balances Token Efficiency and Quality

Cost per correct answer divides logged token spend by grounded, correct responses

The metric has a numerator and a denominator, and each one needs its own discipline.

The numerator is RAG cost per query summed across a test run. For every question, log the input tokens (system prompt, retrieved context, question) and the output tokens. Then multiply each by your contracted per-token price. Use your own rate rather than a list price, because enterprise agreements, batch discounts and cached-input pricing vary. Likewise, if a pipeline carries upfront costs such as embedding or graph construction, amortize them over the queries you expect and add them in.

The denominator, by contrast, counts only answers that pass two gates. The first is correctness: does the answer match what a domain expert marked as right? The second is grounding, scored with RAGAS-style metrics. Faithfulness measures whether the claims in the answer are supported by the retrieved context. Context precision measures whether the retrieved chunks are relevant and ranked near the top. As a result, an answer that is correct by luck but unsupported by its context fails the second gate. Over enough questions, that failure mode predicts hallucinations.

Because both gates depend on what counts as right, they need a fixed, expert-labeled question set drawn from real user queries, including the awkward multi-hop ones. Keep that set frozen across every pipeline you compare. Our benchmarking of enterprise retrieval follows the same principle: retrieval accuracy means little unless it is measured against questions the business actually asks. From here on, correctness together with faithfulness and context precision are the RAG evaluation metrics this article refers to.

RAG vs long context: a 26x token bill buys roughly eight points of correctness

With the metric defined, the benchmark gives enough data to work it through. Treat semantic RAG as using T tokens per query at your rate P, so long-context prompting uses 26T. Dividing each pipeline's cost per query by its correctness gives the cost of each correct response.

The per-model splits point the same way. In the benchmark's per-model results, the mini model scored 75.93% with long context, against 67.28% for semantic RAG and 62.35% for keyword RAG. With the nano model, the scores were 70.37%, 63.58% and 59.26%. However, the sources publish no token ratio for keyword RAG and no figures for hybrid or GraphRAG on this benchmark, so the table marks those rows for you to measure.

Taken per answer, long context costs about 23 times more than semantic RAG for each correct response. That ratio holds at pilot and production volumes, but the absolute gap compounds. At production scale, long context does deliver more correct answers in total. Yet it spends the entire 26x premium to close a gap of under eight points, so each additional correct answer costs far more than an average RAG answer. A pilot of a thousand questions hides this, because the absolute spend is small enough to absorb. A production rollout does not.

That arithmetic, however, assumes every wrong answer costs the same, and in regulated work it rarely does. If an unfaithful answer about a batch record or a safety procedure triggers expert review or a compliance finding, the token savings disappear. The cheapest pipeline on paper then stops being the cheapest in practice.

So set the order of constraints explicitly. First, fix a faithfulness floor that your risk and quality teams sign off on. Then choose the pipeline with the lowest per-answer cost among those that clear it. Under that rule, eight points of correctness can justify a 26x token bill when a single wrong answer costs more than the premium paid to avoid it. The same eight points are worthless when a wrong answer only costs a user a second query.

Corvic AI is built for this ordering. It turns complex, multimodal data into grounded insights for regulated and technical industries, so grounding comes first and token minimization second. For example, in one Corvic AI deployment, a global pharmaceutical team cut analysis time by 80%. A saving like that holds only when the answers are trustworthy enough to skip manual re-checking.

Graph-scoped retrieval earns its indexing cost on multi-hop questions

Long context spends its extra tokens at query time, whereas a knowledge graph shifts spending to indexing. That shift is the main difference between RAG and GraphRAG. Vector RAG is cheap to index and pays at query time, often by retrieving more chunks to cover a question that spans several records. GraphRAG, by contrast, pays upfront to extract entities and relationships. In return, it can retrieve a tighter, connected slice of context for questions that chain across documents.

The quality case for graphs is strongest on complex reasoning. A 2026 study of pruned-graph retrievers found that two variants improved complex-reasoning accuracy over HippoRAG2 by about 9 to 11 percentage points on the GraphRAG-Benchmark Medical and Novel subsets. The cost case is less settled, though. A 2026 comparison of basic, graph-based and agentic RAG does not report comparative token-use or indexing-cost figures. You therefore have to measure amortized indexing cost yourself and add it to the numerator.

In practice, then, the graph pays off only when your question mix is genuinely multi-hop, such as tracing a deviation through equipment logs, maintenance notes and inspection images. Causal factor investigation across multimodal records is that kind of workload. For single-document lookups, by contrast, the indexing spend rarely earns back its cost. Because Corvic AI supports knowledge graph construction and GraphRAG over multimodal data, you can split your frozen question set into single-hop and multi-hop subsets. You can then score graph-scoped retrieval against vector RAG on each subset separately.

Reduce RAG token usage one lever at a time, and rerun the harness when prices or models change

Once you have a baseline, improving RAG token efficiency becomes a matter of pulling levers and checking what breaks. Every lever cuts tokens by removing something, and what it removes can hurt quality. The benchmark above does not measure reranking, compression, chunk-size tuning or semantic caching, and no public study gives reliable percentages for them. Treat the directions below as hypotheses to A/B test against your baseline.

Change one lever at a time and rerun the full question set. Keep a change only if the per-answer cost falls while faithfulness stays above your floor. Query volume, however, changes which lever matters most. In high-volume client support workflows, for example, many questions repeat. Caching then often outweighs every retrieval tweak, because an answer served from cache costs almost nothing in generation tokens.

For those comparisons to stay valid, the scoring itself has to be repeatable. Freeze and version the labeled question set. Log input and output tokens per query for every pipeline, and store the faithfulness, context precision and correctness scores alongside them. Then recompute the metric whenever your contracted price changes, a new model ships or the corpus shifts. Promote a pipeline to production only when it clears the faithfulness floor at the lowest cost per correct answer.

A platform such as Corvic AI can host this harness, because it composes vector RAG, hybrid, GraphRAG and long-context variants without glue code or data migration. As a result, each rerun scores the same documents in place. The Corvic AI case studies show the approach in production. To score your own pipelines, apply for Pioneer Access at corvic.ai.