Cutting LLM Costs
without compromising quality

Where the cost actually comes from

LLM billing is by tokens — units of text. You pay separately for input (what you send the model: prompt, context, documents) and for output (what the model generates), and output is always far more expensive than input. Three facts that explain why the bill balloons:

• The input repeats — every request re-sends the same system prompt, the same instructions and the same documents. You pay for them each time again.
• The context grows — in a conversation or an agent, the history accumulates. The 20th request costs several times the first because it carries everything before it.
• Too large a model — you use the strongest model even for simple tasks that a model 5–25× cheaper would do excellently.

Each of these problems has a direct fix. We will go through them in order of impact.

Prompt Caching — the highest-impact technique

This is the biggest saving, and most people do not use it. Prompt caching lets the AI provider "remember" a fixed part of the prompt between requests. Instead of re-processing the 2,000 tokens of the system prompt and documents every time, the model loads them from the cache — at a dramatically reduced price (a token read from cache costs about a tenth of a regular input token).

The mechanism is based on prefix matching: the provider recognizes that the start of the prompt is identical to a previous request and loads it from cache. The processing order is fixed — first the tool definitions, then the system prompt, then the messages. Therefore:

Practical key points: there is a minimum number of tokens for a cache to activate at all (around 1,024 tokens — shorter than that simply will not cache), the cache has a short lifetime (it resets after a few minutes of no use, unless refreshed), and the number of cache "breakpoints" per request is limited. The real saving comes when the same large prefix recurs across many requests — for example a chatbot with a long system prompt, or RAG with the same context documents.

How not to break the cache by accident

This is where most people fail: the cache works by an exact match of the prefix. Any change of a single byte at the start disqualifies everything after it. These are the cache’s "silent killers":

• A dynamic timestamp in the prompt — injecting datetime.now() into the system prompt changes it on every request and breaks the cache completely. If you need the time, put it at the end.
• Unsorted JSON — if the key order in JSON changes between requests, the prefix differs. Always sort.
• A changing tool list — tools added/removed or reordered disqualify the cache. Keep a fixed order.
• A unique ID at the start — a request or session ID at the start of the prompt = a new cache every time.

Batch Processing — 50% off non-urgent work

Not every request needs an immediate answer. If you process 10,000 documents overnight, classify a dataset, or generate product descriptions — that is asynchronous work. Most providers offer a Batch API that processes such requests within a time window (usually up to 24 hours) for a 50% discount off the regular price.

The rule is simple: anything that does not need an answer within a second — into a batch. A user waiting on a chat needs real time; an overnight report run does not. Half the bill on all that work simply disappears.

Model choice & context

The most expensive mistake is using the strongest model for everything. The gap between a flagship model and a small one is 5× to 25× in price. The vast majority of tasks — classification, extraction, short summarization, routing — do not need the strongest model.

• Route by complexity — simple tasks to a small, cheap model, complex tasks to a flagship. You can even let a small model do the routing.
• A local model for high-volume tasks — for recurring tasks you can run a local model (Llama/Qwen) at zero per-request cost.
• Trim the context — in agents and long conversations, do not drag infinite history. Summarizing or clearing old context saves tokens on every request.
• Short output — output is several times more expensive than input. Ask the model for concise answers and cap max_tokens accordingly.

The savings checklist