What changed: Updated 18 August 2026: added the three-state planning mode, the planner timeout and its fail-open reason codes, plan-honoured accounting, and why eligibility, ranking and fallback are enforced as separate decisions. This is implementation detail that did not exist when the guide was first published.

How to Write an AI Model Routing Policy for Real Workloads

Inconsistent AI model selection across applications introduces significant operational risk.

As AI agents move from pilot projects to core enterprise functions, organizations face a critical challenge: ensuring model choices align with business priorities. Applications often hard-code model providers, leading to fragmented infrastructure. This makes it difficult to manage shared quality, cost, latency, access, and fallback rules. The result is inconsistent performance, unexpected costs, and challenges reconstructing failures end to end. This fragmentation also means tenant isolation and capacity controls often arrive late, after problems emerge.

The Challenge of Inconsistent AI Model Selection

When engineering teams hard-code AI model providers directly into their applications, they create silos. Each application might select models based on immediate developer preference or a single-use case. This approach hinders a unified strategy for model consumption. Critical factors like cost, performance, and data governance become inconsistent across the organization.

For instance, one team might optimize for speed, while another prioritizes cost. Without a centralized routing policy, these individual choices can lead to a diverse, unmanageable agent infrastructure. This fragmentation makes it nearly impossible to gain a comprehensive view of model usage, troubleshoot issues efficiently, or enforce enterprise-wide standards. Debugging failures across multiple, independently chosen models and providers adds complexity and time to reconstruction efforts. Teams spend more time identifying the source of an error than resolving it.

Building Your AI Model Routing Decision Tree

A structured AI model routing policy begins with a decision tree. This framework prioritizes business-critical factors before addressing preferences or price. Starting with workload criticality and data constraints ensures that core business requirements drive model selection, establishing a resilient and compliant AI environment.

Step 1: Workload Criticality

Define the importance of each AI workload. Is it a mission-critical operation, like fraud detection, or a less critical task, such as content summarization? Critical workloads demand models known for high reliability, robust security, and predictable performance. Non-critical tasks might tolerate models with different cost or latency profiles.

This initial classification forms the top branch of your decision tree. It immediately filters eligible models to those that meet the baseline stability and performance required by your most important business processes.

Step 2: Data Constraints and Residency

Understanding the sensitivity and location requirements of your data is paramount. Certain workloads may process highly confidential information or be subject to specific regulatory compliance, such as GDPR or HIPAA. This dictates where data can be processed and, by extension, which model providers are eligible. Data residency rules, for example, might restrict model usage to providers operating within specific geographic regions or requiring on-premise deployment. Explicitly addressing these constraints early ensures compliance and mitigates risk.

Step 3: Performance (Latency and Throughput)

After criticality and data constraints, consider performance needs. How quickly must a response be generated? What volume of requests must the model handle? Real-time user interactions require low-latency models, while batch processing can tolerate higher latency but may demand higher throughput. Different models and providers offer varying performance characteristics. Aligning these to your workload's specific needs avoids bottlenecks and ensures a smooth user experience. Microsoft guidance for scalable AI agent solutions connects model and orchestration choices to business demand, cost visibility, governance, and long-term operating efficiency. Architect scalable and cost-efficient AI agent solutions on Azure - Training | Microsoft Learn.

Step 4: Cost Considerations

Cost optimization is crucial for sustainable AI operations. Model pricing varies significantly by provider and usage. Factors include token count, model size, and API call frequency. An effective routing policy accounts for these variables, directing workloads to the most cost-effective model that still meets quality and performance requirements. This often involves dynamic routing based on real-time pricing or predefined budget thresholds. AWS guidance recommends cost-aware model selection, efficient context handling, and explicit cost-quality telemetry rather than choosing models independently from workload design. Reasoning and execution cost optimization - Agentic AI Lens.

Step 5: Model Quality and Capabilities

Evaluate models based on their accuracy, precision, and specific feature sets relevant to the task. Does a model excel at summarization, code generation, or complex reasoning? A routing policy can direct specific query types to models best suited for those tasks. This ensures optimal output quality and leverages specialized model strengths without sacrificing overall efficiency or incurring unnecessary costs on over-specified models.

Step 6: Fallback and Resilience

No system is infallible. A robust routing policy defines clear fallback mechanisms. What happens if a primary model provider experiences an outage or fails to respond? The policy should specify alternative models or providers, and under what conditions a fallback is permitted or even prohibited. For highly sensitive workloads, a fallback might involve a human-in-the-loop or a more conservative, pre-approved model. For less critical tasks, a simple switch to a different provider might suffice. This pre-planned resilience is essential for maintaining business continuity and trust in your AI agent platform.

Addressing Pain Points with a Governed Policy

Implementing a governed AI model routing policy directly addresses key operational pain points in enterprise AI adoption.

The issue of "Agent infrastructure is fragmented across teams" diminishes when a central policy dictates model access and usage. Instead of independent model choices, teams operate within a shared, standardized framework. This reduces redundant efforts and consolidates management overhead.

"Failures are difficult to reconstruct end to end" becomes a less daunting challenge. With a routing policy, every model call is logged and attributed to a specific policy rule and workload context. This transparent audit trail significantly simplifies debugging and root cause analysis, enabling faster problem resolution across the entire AI agent platform.

"Tenant isolation and capacity controls arrive late" is a common issue with ad-hoc scaling. A routing policy, by design, can incorporate these controls from the outset. Policies can define resource quotas per tenant or workload class. This proactive approach prevents resource contention and ensures predictable performance, even as your AI ecosystem grows.

A Representative Operating Scenario

Consider an enterprise running a multi-tenant AI agent platform. One tenant uses agents for internal document summarization (low criticality, public data). Another tenant employs agents for real-time customer support interactions involving sensitive personal data (high criticality, private data, low latency required). A third tenant utilizes agents for financial fraud detection (very high criticality, highly sensitive data, strict latency, and audit requirements).

Without a routing policy, each tenant might hard-code different models. This creates a messy environment where compliance is hard to verify, costs are unpredictable, and performance varies widely. For instance, the fraud detection agent might accidentally use a cheaper, less secure model meant for summarization, leading to a severe data breach.

With a routing policy, the system first identifies the tenant and workload criticality. Fraud detection workloads are routed to highly secure, high-reliability models that meet strict compliance standards and low-latency needs. Fallback to any public model is prohibited. Customer support workloads are routed to models with strong privacy features and moderate latency, with fallback allowed only to pre-approved alternative secure models. Document summarization, being less critical, can leverage more cost-effective models, potentially with broader fallback options. This structured approach ensures every workload uses the right model under the right conditions.

The BlueBear Approach to AI Model Routing

BlueBear, an AI agent platform, offers a distinctive approach to model routing through its governed agent runtime and MCP gateway. Instead of simply providing a proxy, BlueBear fundamentally associates model eligibility and fallback with workspace policy, credits, and workload context. This integration elevates routing from a technical detail to a core operational capability.

The BlueBear MCP gateway acts as an intelligent intermediary. It doesn't just pass requests; it evaluates them against a predefined policy. This policy considers the workload's criticality, data sensitivity, assigned budget (credits), and the specific context of the workspace initiating the request. For example, a workspace designated for "high-security data" will automatically have its model choices restricted to providers meeting those security standards, regardless of the individual agent's configuration.

This approach differs significantly from simple model-routing tools that focus only on load balancing or cost optimization at a superficial level. BlueBear embeds policy enforcement directly into the runtime environment. This ensures that granular controls for tenant isolation, capacity, and compliance are inherent to every model interaction. It's not an add-on; it's how the system operates from the ground up. This reduces the risk of non-compliant model usage and simplifies the audit process, ensuring governance is baked in, not bolted on.

BlueBear Model Routing Policy Specification

The current BlueBear frontend contract makes routing policy concrete rather than aspirational. The fields below are taken from the policy payload assembled by buildModelRoutingUpdatePayload; they are implementation fields, not proposed marketing terminology.

Machine-readable example

This illustrative payload uses the real BlueBear field names. The values are an example for policy review; they are not presented as production measurements.

{
  "mode": "enforced",
  "optimizationGoal": "balanced",
  "dataResidency": "us",
  "allowedBackends": ["approved-primary", "approved-fallback"],
  "blockedModels": ["unapproved-model"],
  "complianceClass": "internal",
  "latencyTier": "interactive",
  "costTier": "standard",
  "fallbackPolicy": "approved-backends-only",
  "auditMode": "full",
  "constraints": {
    "maxLatencyMs": 3000,
    "maxCostUsd": 0.08,
    "minConfidence": 0.85
  },
  "userControls": {
    "allowOptimizedRouting": false
  }
}

Evaluation order

1. Reject models outside the allowed backend, blocked-model, residency, or compliance boundary.
2. Apply latency, cost, and confidence constraints to the remaining candidates.
3. Rank eligible candidates against the declared optimization goal.
4. Execute the selected route and preserve the effective policy with the request evidence.
5. Apply fallback only when the declared fallback policy allows it; otherwise stop and surface the failure.

This separation matters: eligibility is a governance decision, ranking is an optimization decision, and fallback is an exception-handling decision. Combining them into one opaque score makes incident reconstruction and policy review harder.

Diagnostic Checklist for Your Current AI Model Routing

To evaluate your current AI model routing workflow, consider the following:

• Are model choices consistently aligned with the criticality of each workload?
• Do you have clear, enforced rules for data residency that govern which models can process sensitive information?
• Can you easily track and attribute model costs to specific teams, projects, or tenants?
• Is your AI infrastructure fragmented across multiple teams using different, uncoordinated model selection methods?
• How quickly and completely can you reconstruct the end-to-end flow of a failed AI agent interaction?
• Are tenant isolation and capacity controls integrated into your model selection process from the start, or are they afterthoughts?
• How easy is it to swap a model or provider based on performance, cost, or availability without rewriting application code?
• Is your fallback logic standardized and transparent across all AI agent workloads?

Next Steps: Evaluate Your Workflow

Effective AI model routing is not about adding another tool; it is about refining your operational workflow. Before implementing new solutions, assess your existing practices. Understand where model selection introduces inconsistencies, cost overruns, or compliance risks. Use the decision tree framework to identify gaps in your current approach. A well-defined policy ensures your AI agents operate predictably and efficiently. Begin by drafting routing rules for three distinct workload classes within your organization and state explicitly when fallback to alternative models is prohibited for each.

Implementation detail added since this guide was written

The policy fields above describe what a routing decision should express. This section describes how one is actually built, using BlueBear's own gateway as the worked example. The identifiers are quoted from src/services/modelRouting/modelRouting.service.ts; they are contracts in code, not measurements of a fleet, and route planning ships disabled by default in our deployment configuration.

Routing has three states, not two

export type RoutePlanningMode = "off" | "shadow" | "active";

With only off and active, the first time a router's judgement meets production traffic is also the first time it changes a customer's answer. Those are two separate risks and a policy should let you take them separately. Shadow computes the recommendation and records it while execution continues unchanged — with one caveat worth stating, because it inverts the usual assumption: shadow is also where a canary lives, so a configured canary can cause a small, deterministic, capped slice of shadow traffic to execute the recommendation. "We are only in shadow" is not by itself a guarantee that nothing can change. Shadow-mode routing covers the mechanics.

The planner must not become the latency problem

const DEFAULT_ROUTE_PLAN_TIMEOUT_MS = 750;

This is a client-side abort on exactly one HTTP call — the request to the planner — not an end-to-end routing service level, and it does not bound the model call that follows. What matters more than the number is the failure direction: it fails open, to the model the caller already wanted rather than to a hardcoded default, and every failure carries a distinct reason. A planner timeout records route_plan_timeout; a non-2xx records route_plan_http_<status>; a missing planner URL records inference_os_base_url_missing. A policy that says only "fall back on failure" cannot distinguish a slow planner from a misconfigured one, and those need different responses.

Record whether the plan was actually honoured

A policy is a statement of intent, and the executing runtime declines it more often than most savings models assume — a warm runtime that cannot switch mid-session, a model deprecated since the catalog snapshot, an unavailable credential, an explicit user override. Every one of those is correct behaviour, and every one of them makes a projected saving evaporate silently.

The remedy is a field, not a dashboard. Our outcome contract carries an explicit boolean and a reason drawn from a closed set: not_active, model_deprecated, model_not_executable, already_requested, no_selected_model, execution_disabled, provider_unavailable. The doc comment above the type states the consequence of omitting it plainly: without it, "a plan the caller declined for safety reports identically to one it executed", so the activation evaluation "counts refusals as acceptances". Plan-honoured accounting works through it.

Why eligibility, ranking and fallback must stay separate

Eligibility is a governance decision — which models are permitted at all for this tenant, this compliance class, this workspace — and it is enforced twice in our implementation, once when the candidate catalog is assembled and again when the returned recommendation is re-validated against the live catalog before use. Ranking is an optimisation decision inside that permitted set. Fallback is exception handling when the ranked choice is unavailable. Collapsing the three into one opaque score is what makes an incident unreconstructable six weeks later, because there is no longer any way to tell whether a model was chosen or merely permitted.

Where to go deeper

• Shadow-mode routing — proving a saving on real traffic before changing anything a customer depends on.
• Did your router honour the plan? — why a savings model that assumes plans are followed is always optimistic.
• Canary rollout for routing — the eight conditions that should stop a canary before it executes.
• Which agent steps need a frontier model — the eligibility input: which models are permitted for which step.
• Multi-model resilience — routing as a reliability control rather than a price control.

Questions people actually search for

what should an llm routing policy contain

Four separable things, and keeping them separable is the whole point. An eligibility boundary — allowed backends, blocked models, data residency, compliance class — which is a governance decision. Constraints such as maximum latency, maximum cost and minimum confidence. A ranking objective. And a fallback policy stating what may happen when the preferred route is unavailable. A policy that expresses all four as one opaque score cannot be reviewed and cannot be used to reconstruct an incident.

what is the difference between eligibility and ranking in model routing

Eligibility decides which models are permitted at all — a governance decision a human makes and writes down. Ranking chooses among the permitted set — an optimisation decision that can be automated. The practical test is whether you can answer "was this model chosen, or merely allowed?" after the fact. If the two are collapsed into one score, you cannot.

should model routing be automatic

The selection can be, but the permission set should not be. Automating ranking inside a boundary a human declared is a bounded risk; automating the boundary itself means a policy change can happen without anyone deciding to make one. It is also worth being precise about what "automatic" means in any product you evaluate: planning a route and executing one are different acts, and a vendor should be able to tell you which of the two it performs.

how do I audit an llm routing decision

You need the effective policy in force, the eligible candidate set with exclusion reasons, the selected model with its confidence, the model that actually executed, and any fallback reason. The fourth is the one most commonly missing, and without it the record describes an intention rather than an event.