LLMs in production: where the real work is
Beyond the demo — what it actually takes to run an LLM-powered feature in production: evaluation, cost control, latency, and failure modes.
A working demo of an LLM feature takes an afternoon. A production-grade one takes a system. Here’s where the real engineering work hides.
Evaluation before everything
If you can’t measure quality, you’re guessing. Build evals early:
- Unit-level: does the prompt produce the expected output for known inputs?
- Pairwise / LLM-as-judge: rank outputs against a rubric.
- End-to-end: trace the full user scenario, not just the model call.
Evals are the tests of the AI world. Skip them and every change is a leap of faith.
Cost and latency are features
A model that’s 5% better but 10× slower and 10× more expensive is often the wrong choice. Track token cost per request as a first-class metric. Cache aggressively. Route simple queries to cheaper models.
Failure modes you’ll actually hit
- Hallucinated tool calls — the model calls a tool with plausible-looking but wrong arguments.
- Context bloat — stuffing the prompt until quality degrades (yes, more context can hurt).
- Silent regressions — a model upgrade silently changes behavior. Evals catch this; vibes don’t.
The takeaway
The LLM is one component. The system around it — evals, cost control, observability, fallbacks — is where a production feature is won or lost.
