It is Thursday afternoon. You share your screen, paste a clean two-sentence question into your new AI agent, and press send. In three seconds, an articulate paragraph appears. Stakeholders are impressed, leadership calls it ready, and you get approval to deploy.
By Tuesday, the support queue is overwhelmed. Real users did not send clean sentences. They pasted forty lines of unformatted email threads with nested signatures. The agent stalled, latency crossed fifteen seconds, token usage drained the monthly API budget in three days, and the model promised two customers a refund policy that does not exist. (This example is illustrative, but every symptom happens to teams shipping their first AI product.)
The demo was not a deliberate lie. It worked because prototypes run in an artificial vacuum.
Why Demos Lie
Demos succeed because they operate in an environment arranged for success. Four unspoken assumptions create that false confidence.
Clean input. During a demo, the person typing knows how the system works. They write clear, grammatical queries with single intents. They never submit empty fields, slang, or malformed attachments.
One patient user. When testing a prototype, you tolerate a five-second wait without irritation because you know the server is processing. Real users do not wait; after two seconds without feedback, they assume the screen froze and tap submit repeatedly.
No cost pressure. Running thirty test queries through a frontier model costs pennies. Nobody monitors token spend when two engineers test locally. In production, thousands of active sessions passing repetitive context histories quickly exhaust budgets.
No measurement. A prototype passes when its output looks reasonable to the person demonstrating it. Without systematic benchmarks, you cannot determine whether a prompt revision fixed a flaw or silently broke three core workflows.
Here are the five points of failure that surface once real traffic arrives, along with the engineering patterns required to fix them.
1. Messy Real Inputs
What It Looks Like in Production
In production, backend logs fill with unhandled exceptions, malformed JSON errors, and confused replies. Users submit empty messages, paste forwarded email threads filled with legal disclaimers, write in internet slang, or ask four unrelated questions in one prompt.
The agent either answers only the opening sentence, becomes confused by customer email signatures, or crashes on unprintable characters.
Why It Happens
Prototypes assume user input is clean text ready for an LLM context window. Real-world input is noisy.
Language models distribute attention across all input tokens. When a prompt contains legal disclaimers or conflicting instructions, attention scatters across irrelevant text. Without explicit validation, malformed payloads reach the model directly, wasting tokens and causing non-deterministic failures.
How to Fix It
Treat user input like an unsanitized SQL query. Construct an intake pipeline that cleans and validates payloads before the model sees them:
- Collect real inputs early: Save incoming customer support logs, sales inquiries, and beta feedback to discover how users actually phrase requests.
- Build a test set from real data: Turn the most difficult production queries into an automated regression suite.
- Normalize and validate: Enforce character limits, strip email boilerplate, normalize whitespace, and split multi-intent queries into discrete sub-tasks.
import re
from pydantic import BaseModel, Field, field_validator
class SanitizedInput(BaseModel):
raw_query: str = Field(..., max_length=2000)
cleaned_query: str = ""
@field_validator("raw_query")
@classmethod
def reject_empty(cls, value: str) -> str:
trimmed = value.strip()
if not trimmed:
raise ValueError("Query cannot be empty or pure whitespace.")
return trimmed
def sanitize(self) -> "SanitizedInput":
text = re.split(r"(?i)(---|From:|Sent from my|On .* wrote:)", self.raw_query)[0]
text = re.sub(r"[\x00-\x1f\x7f-\x9f]", "", text)
self.cleaned_query = re.sub(r"\s+", " ", text).strip()
return self
Real Example
We deployed a customer support triage agent for an e-commerce client that handled shipping inquiries. On day two, a customer forwarded a support ticket chain containing four previous agent replies, an automated delivery notice, and an 8-line legal disclaimer in the email footer. The agent parsed the disclaimer instead of the customer’s actual question, saw the phrase “all sales are final under statutory regulation 4B,” and told the customer their pending order could not be cancelled. We immediately introduced a preprocessing pipeline that strips quotation markers (>), truncates forwarded reply headers, and extracts only the newest message block before executing the prompt.
2. Latency Kills Trust
What It Looks Like in Production
Users stare at a static spinner for eight to twelve seconds. Assuming the interface is unresponsive, they refresh the page or resubmit their query. Each retry triggers a duplicate execution, creating server backlog and exhausting API rate limits.
On voice agents, high latency is fatal: pauses longer than 800 milliseconds feel unnatural, while delays beyond two seconds cause callers to speak over the bot, triggering interruption loops.
Why It Happens
Demos frequently chain multiple synchronous LLM operations: one call to classify intent, a second to generate search queries, a third to summarize documents, and a fourth to draft the response.
If each API hop takes 2.2 seconds, cumulative latency quickly exceeds ten seconds. Waiting for the complete response before returning any tokens forces users to experience the worst-case completion time.
How to Fix It
Treat latency as an architectural constraint from the start:
- Stream responses: Use Server-Sent Events (SSE) or WebSockets to display tokens immediately. Optimize for Time to First Token (TTFT), targeting under 1,000 ms for text and under 400 ms for voice.
- Cut chained calls: Combine intent classification and structured extraction into a single prompt whenever possible.
- Parallelize independent calls: Use asynchronous tasks to execute retrieval queries and user profile lookups concurrently.
- Set hard timeouts and fallbacks: If a deep retrieval call exceeds its time budget, cancel it and fall back to a deterministic answer.
import asyncio
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
app = FastAPI()
async def stream_tokens(prompt: str):
for token in ["Your ", "account ", "status ", "is ", "active."]:
await asyncio.sleep(0.06)
yield f"data: {token}\n\n"
@app.post("/agent/chat")
async def chat_handler(query: str):
context_task = asyncio.create_task(get_context(query))
user_task = asyncio.create_task(get_user_metadata())
try:
context, user = await asyncio.wait_for(
asyncio.gather(context_task, user_task), timeout=1.2
)
except asyncio.TimeoutError:
context, user = "Fallback context", {}
prompt = f"Context: {context}\nUser: {query}"
return StreamingResponse(stream_tokens(prompt), media_type="text/event-stream")
async def get_context(q: str): await asyncio.sleep(0.25); return "Context"
async def get_user_metadata(): await asyncio.sleep(0.15); return {"id": 101}
Real Example
While building an inbound appointment scheduling voice agent for a dental clinic group, our prototype used sequential steps: speech-to-text, an LLM call to classify the intent, a second LLM call to generate a calendar lookup tool payload, an API call to the EHR system, and a final LLM response generation step before feeding text-to-speech. The Time to First Audio Byte was 3.8 seconds. Callers repeatedly said “Hello? Are you still there?” and hung up. We eliminated the standalone intent classifier, unified tool generation into a single structured prompt, executed the EHR availability lookup concurrently with patient profile retrieval via asyncio.gather(), and streamed the first token chunk directly to the streaming TTS endpoint. Time to First Audio dropped from 3,800 ms to 680 ms, eliminating caller drop-off immediately.
3. Cost Creeps Up Quietly
What It Looks Like in Production
Your first monthly provider invoice arrives at three to four times your budget. As active users increase, inference costs outpace product revenue.
You discover that your multi-turn chat endpoint resends the complete conversation history on every message, making turn ten ten times more expensive than turn one. Uncapped retry loops and bot traffic consume credits without generating business value.
Why It Happens
During development, using a top-tier frontier model for every task is easy. It handles intent classification, formatting, data parsing, and copy generation without requiring prompt optimization.
In production, using a heavy reasoning model for simple boolean routing is unnecessary overhead. Adding thousands of static prompt tokens and running unbounded chat histories compounds costs quadratically over long sessions.
How to Fix It
Implement financial observability and model tiering across your application:
- Track cost per request: Record prompt tokens, completion tokens, and dollar cost on every trace.
- Route by complexity: Use small, specialized models for intent classification and entity parsing. Reserve frontier models for complex reasoning.
- Cache repeated queries: Store exact matches in Redis and evaluate semantic caching for high-frequency documentation questions.
- Prune conversation histories: Summarize or window earlier conversation turns instead of appending raw message arrays indefinitely.
import hashlib
from typing import Literal
def hash_query(text: str) -> str:
return hashlib.sha256(text.strip().lower().encode()).hexdigest()
async def run_cost_aware_pipeline(query: str, redis_conn) -> str:
cache_key = f"cache:{hash_query(query)}"
cached = await redis_conn.get(cache_key)
if cached:
return cached.decode("utf-8")
intent = await classify_intent_cheap(query)
if intent == "faq":
response = await query_small_model(query)
await redis_conn.setex(cache_key, 86400, response)
else:
response = await query_frontier_model(query)
return response
async def classify_intent_cheap(q: str) -> Literal["faq", "complex"]: ...
async def query_small_model(q: str) -> str: ...
async def query_frontier_model(q: str) -> str: ...
Real Example
On an internal knowledge agent serving 350 employees across engineering and operations, our first month’s bill reached $4,200 against an anticipated $800 budget. An audit revealed two culprits: the client interface was sending the entire conversation history (often 25+ turns containing pasted log files) on every single query, and we were routing trivial questions like “What is the Wi-Fi password?” and “Where is the PTO form?” to a frontier reasoning model. We implemented an exact-match Redis cache for identical questions with a 24-hour TTL, added semantic caching via embedding similarity for FAQ queries, and routed all requests through a lightweight classifier that directed 72% of standard operational queries to a smaller, sub-dollar-per-million-tokens model. The following month’s bill dropped to $940—a 77% cost reduction—with no measurable decrease in user satisfaction scores.
4. Hallucinations (Confident Wrong Answers)
What It Looks Like in Production
A customer asks your support bot about return terms, and the bot replies: “You can return any product within 60 days for a full cash refund, no receipt required.” Your actual policy is 14 days with original packaging. The customer demands that your team honor the promise.
In an internal workflow, an automation agent invents a database column name, causing scheduled ETL processes to crash.
Why It Happens
Language models do not store facts like databases. They are statistical engines trained to generate plausible sequences of words. When context is missing, the model predicts the most probable next words based on broad training data.
Because prototypes are usually tested with general questions, they sound accurate. When production users ask for specific internal policies or inventory details, an ungrounded model will fabricate plausible-sounding answers with complete confidence.
How to Fix It
Anchor generation to verified data and validate outputs programmatically:
- Ground in retrieved data: Use Retrieval-Augmented Generation (RAG) to inject relevant internal documents into the prompt context before answering.
- Authorize “I don’t know”: Explicitly instruct the model to state that it lacks information if the context does not contain the answer, rather than guessing.
- Require source citations: Require the model to link every factual assertion to a specific chunk or document identifier.
- Validate structured schemas: Force tools and integrations to return typed Pydantic models. Reject responses that fail schema validation.
from pydantic import BaseModel, Field
from typing import List
class GroundedResponse(BaseModel):
has_sufficient_context: bool = Field(
..., description="False if retrieved context lacks data to answer."
)
answer: str = Field(
..., description="Direct answer based only on retrieved context, or admission of missing info."
)
cited_chunk_ids: List[int] = Field(
default_factory=list, description="IDs of context chunks supporting the answer."
)
def is_valid_grounding(self) -> bool:
if self.has_sufficient_context and not self.cited_chunk_ids:
return False
return True
Real Example
During a pilot deployment of an internal HR policy agent, an employee asked if unused vacation days rolled over into the next fiscal quarter. The company handbook had no explicit mention of quarterly rollovers—it only specified annual forfeiture. Because the retrieval step returned no matching chunks, the ungrounded model filled the void by stating: “Yes, you can roll over up to 5 unused vacation days into Q1 by submitting Form HR-12.” There was no Form HR-12. Employees began filing non-existent tickets with HR. We fixed this by adding strict negative constraint instructions in the system prompt, enforcing a mandatory has_sufficient_context boolean field via Pydantic structured outputs, and routing queries that fail grounding to a human escalation queue rather than generating speculative text.
5. Missing Evals
What It Looks Like in Production
An engineer updates a system prompt to fix a bug reported by sales. Two days later, customer success discovers that the agent stopped including links in all responses.
Nobody caught the regression because testing consisted of typing a few prompts into an internal playground. You want to migrate to a newer model to cut costs, but no one knows if it will maintain output quality across your existing user journeys.
Why It Happens
Software teams would never deploy thousands of lines of backend code without automated tests. Yet teams routinely ship non-deterministic prompts based entirely on visual inspection.
Because LLM outputs are unstructured and varied, traditional unit testing feels awkward. Without a standardized test set and automated evaluation harness, prompt changes are guesswork: fixing one bug silently breaks two others.
How to Fix It
Build a continuous evaluation system that runs like a standard test suite:
- Start with 20 real cases: Collect twenty real questions where your prototype previously failed, define acceptable answers, and commit them as code.
- Grow a golden dataset: Add interesting queries, edge cases, and edge failures from production logs to an expanding test set.
- Use LLM-as-a-judge with spot checks: Use a powerful model to score responses against a strict rubric for accuracy, tone, and format compliance. Manually review twenty graded samples weekly to keep the judge calibrated.
- Run evals in CI/CD: Execute your evaluation runner on every prompt change, tool update, or model switch. Block pull requests that drop quality scores.
import pytest
GOLDEN_TESTS = [
{
"query": "What is your refund policy?",
"expected_facts": ["14 days", "original packaging"],
"forbidden_phrases": ["60 days", "full refund without receipt"]
},
{
"query": "How do I reset my password?",
"expected_facts": ["settings page", "security tab"],
"forbidden_phrases": ["call customer support"]
}
]
@pytest.mark.asyncio
@pytest.mark.parametrize("test_case", GOLDEN_TESTS)
async def test_agent_factual_accuracy(test_case):
response = await run_agent(test_case["query"])
for forbidden in test_case["forbidden_phrases"]:
assert forbidden.lower() not in response.lower(), f"Found forbidden: '{forbidden}'."
for expected in test_case["expected_facts"]:
assert expected.lower() in response.lower(), f"Missing fact: '{expected}'."
async def run_agent(q: str) -> str: ...
Real Example
When migrating a customer-facing sales agent from an older model checkpoint to a newer, faster release, manual testing across ten sample questions suggested everything was working smoothly. Fortunately, we had accumulated a golden dataset of 45 production edge cases—including tax calculation queries, discounted enterprise seat pricing, and international shipping boundaries. When we ran our automated Pytest evaluation harness against the candidate model, the test suite flagged an immediate regression: for queries involving Canadian postal codes, the new model omitted the required GST tax disclaimer in 6 out of 8 regional cases. Catching that regression in CI before deployment saved us from shipping a compliance violation into live customer conversations.
Production-Readiness Checklist
Review these eleven items before promoting any AI agent or LLM pipeline to production:
- Input Sanitization: Empty payloads are rejected, message length is bounded, and control characters are stripped.
- Intent Segregation: Multi-intent queries are parsed and handled before task execution.
- Streaming Responses: User interfaces consume tokens via Server-Sent Events or WebSockets.
- Concurrent Fetching: Context retrievals and database lookups run in parallel using async routines.
- Timeout Safeguards: Hard timeouts and fallback responses protect every external API call.
- Token Observability: Input tokens, output tokens, and dollar costs are tracked per request.
- Model Tiering: Simple classification and extraction run on lightweight models rather than frontier LLMs.
- Context Grounding: Prompts instruct models to rely strictly on provided data and declare missing facts honestly.
- Schema Validation: Tool invocations and structured data outputs are parsed through strict Pydantic schemas.
- Golden Dataset: A version-controlled dataset of 20 to 50 real-world edge cases is maintained in the repository.
- CI Regression Suite: Automated evaluation runs on every prompt, tool, or model adjustment.
Conclusion
When an AI demo fails in production, the instinctive reaction is to blame the model and look for a bigger one. Teams assume that upgrading to the newest frontier release will fix hallucinations, handle messy input, and improve reliability.
In practice, the fix is usually boring engineering. Reliable systems are built with input validation, concurrent network calls, Redis caching, structured output schemas, and automated test suites. The model provides raw reasoning, but standard software engineering provides the architecture that keeps the system stable under load.
If you treat a language model like an all-knowing oracle, it will let you down. If you treat it like an untrusted, non-deterministic microservice with specific latency and cost profiles, you can build systems that deliver value every day.
Which of these five bottlenecks did your team run into first? Send me an email or leave a comment below—I would enjoy hearing what broke in your system and how you resolved it.
Written by Muhammad Faizan Nadeem
Email: imfaizannadeem2@gmail.com
Website: https://faaizan.space
