LLM Caching

Why caching matters

Every LLM call costs money (by tokens) and time. But a large share of calls repeats — the same long system prompt, the same fixed context, or even the same question asked again and again. Caching means: compute once, reuse. The result — cost savings (sometimes tens of percent) and much faster responses.

1. Prompt Caching (Context Caching)

Leading LLM providers let you cache the fixed, long part of the prompt — the system prompt, instructions, a context document that recurs on every call. On subsequent calls, that part isn't recomputed and is billed at a fraction of the price.

When it's gold: a chatbot with a long system prompt, RAG with a fixed document, or an agent that sends the same instructions repeatedly. Example with Anthropic:

import anthropic
client = anthropic.Anthropic()

resp = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    system=[{
        "type": "text",
        "text": LONG_SYSTEM_PROMPT,          # instructions + fixed context
        "cache_control": {"type": "ephemeral"}  # mark for caching
    }],
    messages=[{"role": "user", "content": user_question}],
)
# the first call builds the cache; the following ones use it and are much cheaper

• Put the fixed part at the start (the cache works on a shared prefix).
• The cache is usually short-lived (minutes) — effective for continuous traffic.
• With OpenAI it's usually automatic for long, repeated prompts.

2. Semantic Caching

Here you save the entire call: if a question similar in meaning was already answered — return the cached answer without calling the LLM at all. You use embeddings: compute a vector for the question, search the cache for a question with cosine similarity above a threshold, and if found — return it.

def semantic_cache_get(question, cache, threshold=0.92):
    q_vec = embed(question)
    for entry in cache:                 # in production: a vector DB, not a loop
        if cosine(q_vec, entry["vec"]) >= threshold:
            return entry["answer"]      # cache hit — no LLM call
    return None

answer = semantic_cache_get(q, cache)
if answer is None:
    answer = call_llm(q)                # cache miss
    cache.append({"vec": embed(q), "answer": answer, "q": q})

• When it's great: FAQ, support, common questions that recur in different phrasings.
• The threshold is critical: too high — misses; too low — inaccurate answers. Calibrate (~0.9+).
• Careful with personalization: don't return a cached answer when the answer depends on a specific user/context.

3. Exact-match Caching

The simplest: the cache key = a hash of the full input (prompt + parameters). If exactly the same input recurs — return the stored output. Fast and cheap, but only catches exact repeats. Good for deterministic tasks (temperature 0) that recur exactly.

import hashlib, json
def key(payload): return hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()

k = key({"model":"gpt-5.6","temp":0,"prompt":prompt})
if k in store: return store[k]
out = call_llm(prompt); store[k] = out; return out

Cache invalidation

A stale cache = wrong answers. Manage it:

• TTL: automatic expiry (hours/days) based on how much the information changes.
• Content version: if you updated the knowledge base/prompt — invalidate the relevant cache (e.g. include a version in the key).
• Don't store sensitive/personal information in a shared cache.

Common mistakes

• Semantic cache with too low a threshold. Returns the answer of an "almost similar" question and hurts accuracy.
• Storing user-dependent answers in a global cache — one customer sees another's answer.
• A cache without a TTL. Answers go stale and become wrong.
• Putting the dynamic part at the start. Breaks prompt caching — the fixed part must be the prefix.
• Forgetting to measure hit rate. Without measurement you don't know if the cache even helps.