Orchestrating Multiple LLMs Without Losing Control

When we started routing production voice traffic through more than one LLM provider, the pitch was simple: use whichever model is best suited to the turn. Claude for anything that needs careful instruction-following or longer context, GPT-4 for latency-sensitive turns where its response time had an edge. In practice, the orchestration layer turned out to be the easy 20%. The eval harness that decides whether a change is safe to ship is the other 80%.

Why orchestration alone isn't the hard part

LangChain and LangGraph give you the primitives to wire up multi-model routing quickly: a graph of nodes, conditional edges based on intent classification or confidence scores, and shared state that flows between them. Getting a call routed to the right model based on turn type, conversation stage, or fallback conditions is a few days of work, not a few months.

The properties that actually determine whether this holds up under production load are less visible:

• Fallback paths need to be cheap and fast. If Claude times out mid-call, GPT-4 needs to pick up the conversation state without the caller noticing a beat. That means shared, provider-agnostic conversation state — not per-provider message formats bolted on after the fact.
• Model swaps change behavior in ways that are hard to predict from documentation alone. A prompt tuned against one model's instruction-following quirks will not perform identically against another, even with equivalent system prompts. You find this out in production unless you find it first in evals.
• Cost and latency budgets are per-call, not per-model. Orchestration logic has to reason about a call's remaining budget, not just which model is "best," because the best model for a turn in isolation is sometimes the wrong choice given what's already been spent on that call.

The eval harness is the actual product

We treat our eval harness as a release gate with the same seriousness as a test suite gates a merge. No orchestration change — a new prompt, a new routing rule, a provider version bump — ships without passing it first.

The harness runs three tiers of checks:

1. Golden transcript replay. A curated set of real (anonymized) call transcripts gets replayed against the candidate configuration, and outputs are diffed against known-good baselines using both semantic similarity and task-specific assertions (did it collect the right slot values, did it avoid a disallowed claim, did it hand off correctly).
2. LLM-as-judge scoring on hard cases. For the subset of transcripts involving ambiguity, interruption handling, or edge-case intents, a separate judge model scores the candidate's response quality against a rubric. We deliberately use a different model family for judging than for the primary orchestration path, to reduce the chance that a shared blind spot passes itself.
3. Compliance and safety assertions. Because the platform operates under HIPAA, GDPR, and APP obligations across regions, every candidate run is also checked against a fixed set of hard constraints — no PHI leakage across turns, no data retention violations, correct region-scoped routing. These are binary gates, not scored; a single failure blocks the release regardless of how well everything else performed.

What we'd tell a team starting this today

Build the eval harness before you need it, not after the first regression makes it obvious you do. It is significantly harder to retrofit a golden dataset and judge rubric onto a system that's already live and already has stakeholders who trust its current behavior. Start capturing transcripts and building assertions from the first week production traffic exists, even if the harness itself is rough.

Multi-LLM orchestration is a genuinely good idea for systems that need to balance quality, cost, and latency across different types of work. Just budget for the harness, not only the router.