Most RAG demos look sharp in a notebook and fall apart the first week of production traffic. Latency spikes, retrieval quality drifts, and the wrong model answers with confidence. Teams respond by stuffing more context into the prompt or swapping models mid-sprint and the outage pattern repeats.
We ship RAG as a gated pipeline: retrieve, score, route, generate, then verify. Each stage has a clear failure mode and a fallback that keeps answers useful when the preferred path is slow or low-confidence. That discipline is what separates a demo from a system you can operate on a Friday.
The fix is not a bigger prompt. It is a routing and evaluation system that treats model choice as an operational control.
Gate model choice, don’t guess
Cheap models handle high-volume, low-risk queries. Stronger models unlock when retrieval confidence is thin, the domain is regulated, or the user is mid-workflow. Routing rules are written as score thresholds and policy tags, not as one-off if-statements buried in the handler.
We keep a short allowlist of models per environment and log every route decision with the retrieval score that triggered it. That audit trail is what makes Friday incidents diagnosable instead of folklore.
When a route fires unexpectedly often, the dashboard shows it. Product and ML review the threshold together, usually the retrieval index drifted, not the model itself.
Evals before feature theater
Offline eval sets cover citation faithfulness, refusal quality, and latency budgets. Online shadow traffic compares candidates without exposing users until the gate passes. If an eval regresses, the route rolls back automatically.
We refuse to ship a “better” model that fails faithfulness even if latency looks prettier. Feature theater is swapping models for the release note. Real progress is a gate that keeps retrieval useful under real load.
What we measure in week one
p95 end-to-end latency by route, citation hit rate on a frozen eval pack, and fallback rate under peak. If fallback rate climbs, we investigate retrieval before we buy a larger model.
The result is a system the team can change without rewriting the product surface every sprint and answers that still hold when traffic is ugly.
