Cutting LLM Costs Without Shipping a Worse Agent

The instinctive move when an LLM bill gets uncomfortable is to swap the model for a cheaper one and call it done. Sometimes that’s genuinely fine. Often it isn’t, and the way it isn’t is insidious: the agent still answers, the demo still looks fine, and three weeks later someone notices the classification accuracy quietly dropped four points, or the agent started missing an edge case it used to handle. Nobody connects that regression to the model swap because nothing crashed — it just got worse in a way that doesn’t show up unless you’re specifically measuring for it.

The actual problem with LLM cost optimization isn’t finding techniques that save money — model routing, prompt caching, context trimming, and semantic caching all reliably do. It’s that every one of them is a lever that trades a dollar amount for a quality risk, and most teams pull the lever without ever building the test that would tell them how much quality they actually gave up. This post covers four concrete techniques and, for each one, the specific regression test it needs before it goes anywhere near production traffic.

You’ll learn:

• Why “just use a cheaper model” fails silently instead of loudly, and what to measure instead
• How to build a model router that sends easy calls to a cheap model without guessing
• How prompt caching works mechanically, and why prefix stability is the whole game
• A context-trimming strategy that doesn’t quietly delete the fact your agent needed three turns later
• When semantic caching is worth the staleness risk, and when it silently poisons answers
• A concrete, reproducible methodology for testing each technique before shipping it
• How to combine all four into a single cost-aware call path without stacking their risks

Table of Contents

1. Why “Just Downgrade the Model” Fails Silently
2. Model Routing
3. Prompt Caching
4. Context Trimming
5. Semantic Caching
6. A Reproducible Testing Methodology
7. Putting It Together: A Cost-Aware Call Path
8. At a Glance
9. Wrapping Up

Why “Just Downgrade the Model” Fails Silently

A model swap doesn’t throw an exception when it makes your agent worse. Your latency might even improve. The failure surfaces later, and usually indirectly — a support ticket about a wrong answer, a slightly higher escalation rate nobody’s tracking closely enough to notice for weeks, a structured-extraction pipeline that starts silently dropping an optional field a smaller model is less reliable about populating. None of that trips an alert, because nothing in a typical setup is watching for “did quality change,” only for “did the service stay up.”

That’s the actual argument for building a regression test before touching cost, not after: cost is trivially observable (it’s a number on an invoice) and quality is not, and optimizing the thing you can see while ignoring the thing you can’t is exactly how a cost-cutting change quietly becomes a product regression. Every technique below gets cheaper in a way that’s easy to verify and risks quality in a way that isn’t — unless you build the harness for it up front. If the agent you’re optimizing is graph-based with branching logic rather than a single call, the distinction between a simple chain and something that needs LangGraph is worth having settled first, since where you insert routing and caching differs a fair amount between the two.

Model Routing

Not every call into an agent needs your most capable model. A classification step deciding “is this a billing question or a technical one” and a step drafting a nuanced response to an angry customer are not the same difficulty, and paying frontier-model prices for both is spending money on the easy one for no reason.

The mechanism that actually works is a lightweight classifier — often a small model itself — that scores task difficulty and routes accordingly, rather than a hardcoded per-endpoint mapping that has to be manually maintained as your product changes.

import anthropic

client = anthropic.Anthropic()

def classify_difficulty(query: str) -> str:
    response = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=10,
        messages=[{
            "role": "user",
            "content": f"Classify this support query as 'simple' or 'complex'. "
                        f"Reply with one word only.\n\nQuery: {query}",
        }],
    )
    return response.content[0].text.strip().lower()

def route_and_answer(query: str) -> str:
    difficulty = classify_difficulty(query)
    model = "claude-sonnet-5" if difficulty == "simple" else "claude-opus-5"
    response = client.messages.create(
        model=model,
        max_tokens=1024,
        messages=[{"role": "user", "content": query}],
    )
    return response.content[0].text

The classifier call itself costs something, so routing only pays off when the volume-weighted savings from downgrading a meaningful share of traffic outweighs the classification overhead — in practice this is almost always true once you’re past a small number of daily calls, since the classifier is a tiny, fast, cheap call compared to the generation it’s gating. As of this writing, Claude Haiku 4.5 runs at roughly $1 per million input tokens against Claude Sonnet 5’s roughly $3, so a classification prompt of a few hundred tokens is a rounding error next to what you save by not sending easy queries to a pricier model — but treat those figures as illustrative, not a promise, since list prices change and your own token counts will differ.

The failure mode is a router that’s wrong in the expensive direction — routing something genuinely complex to the cheap tier and getting a worse answer nobody notices because the response still looks plausible. That’s exactly why routing needs the regression harness in the testing section below, not just a spot check of a few examples that happened to route correctly.

Prompt Caching

Prompt caching is the closest thing to a free win on this list, because it doesn’t touch model choice or trim anything — it just avoids re-paying for tokens you’re sending unchanged, over and over, across requests. Claude’s prompt caching works on prefix matching: mark a point in your request with a cache breakpoint, and if a later request shares that exact prefix, the cached portion is billed at a steep discount instead of full input price.

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    system=[{
        "type": "text",
        "text": LONG_SYSTEM_PROMPT_WITH_TOOL_DEFINITIONS,
        "cache_control": {"type": "ephemeral"},
    }],
    messages=[{"role": "user", "content": user_query}],
)

print(response.usage.cache_read_input_tokens)  # tokens served from cache, billed at a discount

The mechanical detail that determines whether this actually works is prefix stability. Caching is a byte-for-byte prefix match — a single changed character anywhere before your cache breakpoint invalidates everything after it. The most common way teams accidentally defeat their own caching is putting something volatile before the breakpoint: a timestamp interpolated into the system prompt, an unsorted dict serialized differently between requests, a request ID baked into instructions that should have stayed generic. response.usage.cache_read_input_tokens is the number to watch — if it’s consistently zero across requests that should be hitting cache, something upstream is silently varying the prefix.

The quality risk here is genuinely low compared to the other three techniques, because caching doesn’t change what the model sees — it changes how that unchanged content is billed. The test you still want is a check that cache-hit and cache-miss responses are behaviorally identical for the same input, which mostly serves to catch bugs in your own caching setup (a breakpoint placed in the wrong spot, content ordering that differs between “tools then system” versus “system then tools”) rather than any inherent risk in the technique.

Context Trimming

An agent’s context grows every turn — more messages, more tool results, more history — and every one of those tokens gets re-sent and re-billed on every subsequent call unless something actively prunes it. Context trimming cuts that regrowth, but it’s the technique most likely to break something subtle, because the token you trim might be the one piece of information a later turn actually needed.

A defensible trimming strategy keeps recent turns verbatim and summarizes or drops older ones, rather than trimming uniformly or by raw token count alone:

def trim_context(messages: list[dict], keep_recent: int = 6) -> list[dict]:
    if len(messages) <= keep_recent:
        return messages
    older, recent = messages[:-keep_recent], messages[-keep_recent:]
    summary = summarize_messages(older)  # a cheap model call, or a rule-based extractor
    return [{"role": "user", "content": f"[Earlier conversation summary]: {summary}"}, *recent]

The failure mode to design against specifically is losing a fact that only matters several turns later — a constraint the user stated early (“I’m allergic to shellfish,” “budget is $500 max”) that a naive recency-based trim silently drops once it ages out of the “recent” window. A summarization step that’s actually instructed to preserve constraints, decisions, and stated facts (not just narrative gist) is meaningfully safer than a hard token-count cutoff, and it’s worth writing that instruction explicitly rather than trusting a generic “summarize this” prompt to prioritize the right details.

Trimming tool results deserves separate treatment from trimming conversation turns — a large JSON payload from an earlier tool call is often safe to compress or drop once the model has already used it to answer, since it’s raw data rather than a stated fact, whereas the model’s own conclusion drawn from that data usually is worth keeping.

Semantic Caching

Semantic caching skips the model call entirely when a new query is judged similar enough to one you’ve already answered — typically via embedding similarity against a cache of past query/response pairs, rather than the exact-match caching that a simple key-value cache would give you.

def semantic_cache_lookup(query: str, threshold: float = 0.92) -> str | None:
    query_embedding = embed(query)
    best_match, best_score = None, 0.0
    for cached_query, cached_response, cached_embedding in cache_store:
        score = cosine_similarity(query_embedding, cached_embedding)
        if score > best_score:
            best_match, best_score = cached_response, score
    return best_match if best_score >= threshold else None

def answer_with_semantic_cache(query: str) -> str:
    cached = semantic_cache_lookup(query)
    if cached is not None:
        return cached
    response = client.messages.create(
        model="claude-sonnet-5", max_tokens=1024,
        messages=[{"role": "user", "content": query}],
    )
    answer = response.content[0].text
    cache_store.append((query, answer, embed(query)))
    return answer

This is the riskiest technique on this list, and it’s worth being direct about why: the whole premise is answering a new question with a response generated for a different question that merely scored high on embedding similarity. Two queries can be semantically close and still have a materially different correct answer — “what’s your refund policy for electronics” and “what’s your refund policy for electronics bought during the holiday sale” will embed close together and can have genuinely different answers. A cache with too low a similarity threshold doesn’t fail loudly; it serves confidently wrong answers that read as completely normal responses.

Semantic caching earns its risk in narrow situations: high query volume with real repetition (an FAQ-style support bot, not an open-ended coding agent), content where staleness is acceptable or the underlying facts change slowly, and — critically — a threshold you’ve actually tuned against labeled examples rather than a value you picked because it “felt strict enough.” Anywhere the exact wording changes the correct answer, this technique should not be anywhere near the call path.

A Reproducible Testing Methodology

Every technique above needs the same basic thing before it goes near production traffic: a fixed evaluation set, a quality metric that isn’t vibes, and a before/after comparison run against both. None of this requires anything exotic — it requires actually doing it, which is the step most cost-cutting changes skip.

Build a fixed test set from real traffic. Pull 100–300 real queries your system has actually handled (anonymized as needed), covering the range of difficulty you see in production — not just the easy cases that make any technique look safe. If you don’t have production traffic yet, hand-write cases that specifically probe the failure modes described above: a multi-constraint conversation for context trimming, a pair of near-duplicate-but-differently-answered queries for semantic caching, a mix of genuinely easy and genuinely hard requests for model routing.

Score with something reproducible, not a single human skim. For tasks with a checkable answer (classification, extraction, routing decisions), exact-match or field-level accuracy against a labeled set is straightforward and cheap to compute. For open-ended generation, an LLM-as-judge comparing the baseline response against the optimized one on a fixed rubric (correctness, completeness, tone) is a reasonable stand-in for human review — but validate the judge itself against a small human-labeled sample first, since a judge model with its own blind spots can rubber-stamp a regression it happens to share.

Run baseline and optimized side by side, not sequentially. Compare the exact same test set against the pre-change and post-change pipeline in the same run, not “last month’s average quality” against “this month’s” — too many other things change between those two windows to attribute a shift to your specific change with any confidence.

Set a concrete threshold before you look at the results, not after. Decide in advance what quality drop is acceptable for what savings — for illustration only, a team might decide that a 2-point drop in accuracy is acceptable for a 40% cost reduction and a 10-point drop is not, whatever the actual numbers turn out to be for their own system. Deciding the threshold after seeing the number is how a marginal regression gets rationalized into “acceptable” after the fact.

Re-run the same test set after any prompt, model, or threshold change. Model routing thresholds, cache similarity cutoffs, and trimming window sizes all drift as a product evolves — the regression test isn’t a one-time gate, it’s something you re-run whenever one of these knobs moves.

Putting It Together: A Cost-Aware Call Path

The four techniques compose, but each one you stack multiplies the number of places a silent regression could be hiding, which is exactly why the testing methodology above needs to run against the combined pipeline, not just each technique in isolation.

def answer_query(query: str, history: list[dict]) -> str:
    cached = semantic_cache_lookup(query)
    if cached is not None:
        return cached

    trimmed_history = trim_context(history)
    difficulty = classify_difficulty(query)
    model = "claude-sonnet-5" if difficulty == "simple" else "claude-opus-5"

    response = client.messages.create(
        model=model,
        max_tokens=1024,
        system=[{
            "type": "text",
            "text": STABLE_SYSTEM_PROMPT,
            "cache_control": {"type": "ephemeral"},
        }],
        messages=[*trimmed_history, {"role": "user", "content": query}],
    )
    answer = response.content[0].text
    cache_store.append((query, answer, embed(query)))
    return answer

The order matters: semantic cache check first (cheapest possible exit), then trim context before routing so the difficulty classifier isn’t scoring against a bloated history, then prompt caching on the stable system prompt regardless of which model tier gets selected. Each layer is a real, independent savings — but each is also a real, independent place a bug or a too-loose threshold can quietly cost you quality, which is exactly the case for running your regression set against this full path, not four separate untested pieces bolted together and assumed to compose safely.

At a Glance

Reading the table left to right roughly tracks savings potential and risk — semantic caching can eliminate a call’s cost entirely, which is also exactly why a bad hit is a wrong answer, not just a worse one. Prompt caching sits at the other end: comparatively small guaranteed savings on stable content, with almost no way to accidentally serve a wrong answer, because it never changes what the model actually sees.

One more lever worth a mention even though it’s not one of the four core techniques: the Batch API, for anything that isn’t latency-sensitive. Batch requests are typically priced well below the standard synchronous rate in exchange for asynchronous processing — a good fit for nightly summarization jobs, bulk classification, or any pipeline where “the answer shows up in a few hours” is an acceptable trade for a substantially lower bill. It doesn’t compose with the other four the way they compose with each other — you either send a request synchronously through the cost-aware path above or asynchronously through batch — but for the subset of your workload that can tolerate the delay, it’s often the single biggest saving available before you touch model choice, caching, or trimming at all.

Wrapping Up

None of these four techniques are questionable engineering — model routing, prompt caching, context trimming, and semantic caching are all standard, well-understood ways to bring an LLM bill down, and most production systems should be using at least the first two. The part that’s optional isn’t the technique, it’s whether you build the regression test before or after you find out it mattered. Cost is the easy number to watch because it’s right there on the invoice; quality only shows up if you build something that measures it on purpose, against a fixed test set, before the change ships and again every time one of the tuning knobs moves. Do that, and cutting costs stops being a bet against your own product quality and starts being an engineering change like any other — measured, reversible, and boring in the best way.