GUIDE: AI RUNNING COSTSHow to reduce LLM token usage and AI running costs

011. Prompt caching

What it is. When requests share a long prefix (instructions, a policy manual, tool definitions), the provider stores the processed prefix and charges far less to reuse it.

When it helps. Assistants with long, stable system prompts, and agents that resend tool definitions every turn.

What the providers say.

• Anthropic: reads cost 0.1x base input (0.05x on Opus 5.5 and Sonnet 5.5); writes cost 1.25x for the default 5-minute lifetime, refreshed free on each hit, or 2x for 1 hour. Minimum 512 tokens on current models. [1]
• OpenAI: automatic. On GPT-5.6 and later, minimum 1,024 tokens; reads 0.1x (0.05x on GPT-6.1 Sol and Terra); writes 1.25x for the default 30 minutes or 2.5x for 12 hours. [2]
• Google: implicit caching on by default, minimum 4,096 tokens on Gemini 3.8 Flash; cached input 10 per cent of base, plus hourly storage for explicit caches. [3][4]

Pitfalls. Caching matches an exact prefix, so a timestamp at the top of the prompt breaks every hit. Put stable content first. Sparse traffic pays the write premium without the reads.

022. Batch and flex processing

What it is. Submit work that can wait asynchronously, at a lower price.

When it helps. Overnight classification, evaluation runs, data enrichment.

What the providers say. Anthropic’s Message Batches API is 50 per cent off all usage, with most batches finishing in under an hour and a 24-hour limit; it stacks with prompt caching, although cache hits in batches are best effort. [5] OpenAI’s Batch API is 50 per cent off with a 24-hour window and up to 50,000 requests per batch, [6] and its Flex tier is priced at batch rates in exchange for slower responses and occasional unavailability. [7] Google’s Batch API runs at 50 per cent of standard cost with a 24-hour target. [8]

Pitfalls. Batches can take the full 24 hours, so never batch work a customer is waiting on.

033. Model routing and cascades

What it is. Send each request to the cheapest capable model. A router decides up front; a cascade tries a small model first and escalates when a check fails.

When it helps. Mixed workloads that are mostly routine.

What the research and providers say. The RouteLLM paper (ICLR 2025) reported cost reductions of over 85 per cent on MT Bench, 45 per cent on MMLU and 35 per cent on GSM8K against using GPT-4 alone, while reaching 95 per cent of GPT-4’s performance. [9] FrugalGPT reported up to 98 per cent cost reduction while matching the best single model on its benchmarks, a best case rather than a typical one. [10] Anthropic’s own guidance is more cautious: “Tune one model before you add a second,” and its measured multi-model results ranged from about a third cheaper at similar quality to cheaper but measurably worse. [11]

Pitfalls. A cascade pays twice for escalated requests, and weak checks let poor answers through. Measure quality on routed traffic.

044. Evaluation-driven model selection

What it is. Choose the model for each task by testing candidates on your own data, then pick the cheapest one that passes.

When it helps. Always, first. Anthropic’s model guide says “Having a good evaluation set is the most important step in the process” and suggests starting with a fast, low-cost model and upgrading “only if necessary”. [12]

Pitfalls. Public benchmarks rarely reflect your edge cases. Re-run evals when a provider ships a new model version.

055. Trim context and retrieve instead of stuffing

What it is. Send only the passages a question needs, found through retrieval (RAG), instead of pasting whole documents into every prompt.

When it helps. Question answering over large document sets, policy assistants, agents with long tool outputs.

What the sources say. Anthropic’s engineering team calls context “a finite resource with diminishing marginal returns” and recommends “the smallest set of high-signal tokens”. [13] Research also found models use information in the middle of long inputs less reliably than at the start or end. [14]

Pitfalls. Retrieval that misses the right passage yields a confident wrong answer, so evaluate retrieval separately.

066. Summarise and compact long conversations

What it is. Replace older turns of a long session with a summary, so each request carries less history.

When it helps. Long chats and agents, where input grows every turn.

What the providers say. Anthropic’s compaction feature “replaces the older turns of a conversation with a summary that Claude writes on the server”, and its context editing can clear old tool results by rule. [15]

Pitfalls. Summaries can drop detail a later turn needs. Compaction also changes the prefix and resets prompt caching, so with heavy caching the saving is smaller.

077. Cap output length and use structured outputs

What it is. Ask for exactly what you need, in a defined format, with a sensible maximum length.

When it helps. Extraction, classification, form filling and any output a program reads.

What the providers say. OpenAI’s Structured Outputs ensures responses “adhere to your supplied JSON Schema”, so there is “no need to validate or retry incorrectly formatted responses”. [16] Fewer retries mean fewer paid calls.

Pitfalls. A limit set too low truncates answers, and on reasoning models it must leave room for reasoning tokens.

088. Tune reasoning effort

What it is. Reasoning models can think more or less before answering. Hidden reasoning tokens are billed as output.

When it helps. Routine, high-volume or latency-sensitive tasks that do not need deep reasoning.

What the providers say. OpenAI: reasoning tokens are “billed as” output tokens, and “Lower effort favors speed and lower token usage.” [17] Anthropic: low effort gives “Significant token savings with some capability reduction”, and “Tuning effort is often a better lever than switching models.” Defaults differ: Claude Opus 5.5 and Haiku 5.5 default to medium, most others to high. [18][12] Neither provider publishes a fixed percentage.

Pitfalls. Effort levels are recalibrated between model versions; re-test when you upgrade.

099. Deduplicate and use semantic caching

What it is. Return a stored answer when a new question matches an old one exactly (deduplication) or in meaning (semantic caching), skipping the model call.

When it helps. FAQ-style traffic, internal help desks, repeated lookups.

What the research says. One 2024 preprint reported up to 68.8 per cent fewer API calls, with hit rates of 61.6 to 68.8 per cent on customer-service-style queries, a single study on one workload. [19] Open-model servers such as vLLM offer a related feature, automatic prefix caching. [20]

Pitfalls. A loose similarity threshold returns the right answer to the wrong question. Do not cache answers that depend on the user, the date or live data.

1010. Fine-tune or distil to a smaller model

What it is. Use a large model’s best outputs to train a smaller model for one specific task.

When it helps. A stable, high-volume task where a small model nearly passes your evals.

What the providers say. OpenAI describes distilling “the results of a large model to create training data for supervised fine tuning”, so that a small model performs “similarly on a specific task to a larger, more costly model”. [21]

Pitfalls. Distilled models generalise poorly beyond their training data; retrain when the task changes, and check provider terms on training with outputs.

Hosted or self-hosted: where the break-even sits

Self-hosting open models swaps per-token prices for a mostly fixed monthly cost. It pays when that cost is lower than the API spend it replaces at equal quality.

Lambda lists these GPU prices as taking effect 16 November 2026. [22] Recheck before you rely on them.

What this means. On cost alone, self-hosting competes best against frontier-priced APIs at sustained volume, where an open model passes the same evals. Against the cheapest hosted tiers, break-even volume is very high. The stronger case is often not price: data that cannot leave your network, regulation, offline sites and predictable latency.