Making AI Faster and Cheaper: Cost and Latency Optimization

Building an AI feature is one thing; running it for thousands of users without a shocking bill or a sluggish app is another. As you saw with multi-agent systems, every LLM call costs money and time — and they add up fast. This lesson is the practical, money-saving side of AI engineering: the concrete levers that make your app faster and cheaper without gutting quality. These are the tricks that separate a demo from a product.

Why LLM apps get slow and expensive

Three things drive cost and latency:

• Tokens: you pay per token in and out (Lesson 20). Long prompts and long answers cost more and take longer.
• Model size: bigger models are smarter but slower and pricier per token.
• Number of calls: agents, retries and multi-step chains multiply everything.

Optimization is mostly about spending those three wisely — using the least model, the fewest tokens, and the fewest calls that still do the job.

Lever 1: Right-size the model (and route)

The biggest mistake is sending every request to the largest, most expensive model. Most queries don't need it. A router (or "cascade") sends easy questions to a small, cheap, fast model and only escalates hard ones to the big model — often cutting costs dramatically with no quality loss where it matters.

Lever 2: Caching (the biggest quick win)

If users ask the same or similar things repeatedly, don't pay to answer twice — cache the result and reuse it. This is often the single fastest, cheapest optimization. Here's the idea in runnable Python:

cache = {}
calls_made = 0

def expensive_llm(prompt):
    global calls_made
    calls_made += 1                      # pretend this costs money + time
    return f"answer to: {prompt}"

def cached_llm(prompt):
    if prompt in cache:
        return cache[prompt], "CACHE HIT (free!)"
    answer = expensive_llm(prompt)
    cache[prompt] = answer
    return answer, "computed"

print(cached_llm("What is RAG?"))   # computed
print(cached_llm("What is RAG?"))   # CACHE HIT — no LLM call
print("Total real LLM calls:", calls_made)   # 1, not 2

Two identical questions, one paid call. At scale, that's a huge saving. (For "similar but not identical" questions, you can cache by embedding similarity — a semantic cache.)

Lever 3: Trim your prompts and context

Every token counts. Cut boilerplate from prompts, and don't stuff the model with more context than it needs — this is exactly why RAG retrieves only the top few chunks instead of dumping whole documents. Shorter inputs mean lower cost and faster responses.

Lever 4: Stream responses

Some latency is unavoidable, but you can change how it feels. Streaming shows the answer token-by-token as it's generated (like ChatGPT typing), so users see progress instantly instead of staring at a spinner. Same total time, far better experience.

Lever 5: Limit output and batch

• Cap output length: ask for concise answers and set a max-tokens limit — output tokens are often the pricier half.
• Batch requests: processing many items in one call (where supported) is cheaper than many separate calls.

Lever 6: Smaller models via quantization & distillation

If you run your own models, two techniques shrink them: quantization (storing weights at lower precision, e.g. 4-bit, for big speed/memory savings with minor quality loss) and distillation (training a small "student" model to mimic a large one). Both let a smaller, cheaper model do the work of a bigger one on a narrow task.

The most important question: do you even need an LLM?

Sometimes the cheapest, fastest AI is not an LLM. A simple rule, a keyword match, a small classic ML model (Lesson 10), or a cached lookup can handle many tasks for a fraction of the cost and latency. Reach for the LLM when you genuinely need its flexibility — not by default.

Your optimization checklist

The engineer's rule: use the smallest model, the fewest tokens, and the fewest calls that still meet your quality bar. Measure, then optimize the biggest cost first.

The optimization that shocked me most was the simplest — adding a basic cache to a repetitive feature cut its cost by more than half overnight, before I'd touched anything clever like model routing. It taught me to always measure where the money and milliseconds actually go before optimizing, because the biggest win is usually more boring than I expect. My advice: profile first, add caching early, and only reach for the fancy techniques once you know they'll move the number that matters.