Reading time: 9 minutes, 2 seconds

I want to start with a question: what percentage of your AI requests actually need your most expensive model?

Think about your application for a moment. Users ask simple questions. Developers run tests. The system generates summaries, rephrases text, answers FAQs, produces draft emails. Then — occasionally — someone asks for a security audit, a production incident analysis, or a deep architectural review.

If you are running all of those through GPT-4o, you are paying the same premium price for “What is HTTP 429?” as you are for “Analyze the root cause of our production outage and provide a remediation plan.” That is like hiring a senior consultant at $500 per hour to answer your phone.

This article is about fixing that. We will build an intelligent model router that sits between your application and the AI models, automatically deciding which model to use for each request — and saving 60-80% of your AI costs in the process.

The code is on GitHub, the Azure setup is scripted, and every number in this article comes from a real working lab.

The Problem Nobody Talks About

AI cost conversations usually focus on prompt engineering — using fewer tokens, caching responses, compressing context. Those are good ideas. But there is a more fundamental question that rarely gets asked:

Do you even need the expensive model for this request?

The reality is that most LLM workloads are a mix of simple and complex tasks:

In a typical application, 60-75% of requests fall into the “simple” category. Yet most teams route everything to the premium model because it is the path of least resistance — one model, one configuration, done.

The cost difference is significant. GPT-4o costs roughly $2.50 per million input tokens. GPT-4o-mini costs $0.15 per million input tokens — that is 16 times cheaper. If you can route 70% of your requests to the cheaper model, your total bill drops by around 65% without touching a single line of your application logic.

That is what model routing does.

What Is Model Routing?

Model routing is a layer between your application and the AI models that inspects each request and decides which model to send it to.

BEFORE:   Your Application  -->  GPT-4o (always, expensive)
AFTER:    Your Application  -->  Router  -->  GPT-4o-mini  (simple requests)
                                          -->  GPT-4o        (complex requests)

The router makes this decision in milliseconds, with no extra API calls, using a set of strategies applied in priority order. The application never needs to know which model answered — it just gets the response.

This is not a new concept in software engineering. Load balancers do this for servers. API gateways do this for services. We are applying the same idea to AI model selection.

The Lab: What We Built

The lab is a complete, working implementation on Azure:

• Two Azure OpenAI deployments: gpt-4o-mini (cheap tier) and gpt-4o (premium tier)
• A routing engine in Python with five pluggable strategies
• A FastAPI service that exposes /chat, /route, /stats, and /health endpoints
• A cost tracker that logs every call to SQLite and reports actual savings vs a premium-only baseline
• Three demo scripts ranging from offline simulations to live Azure calls

Everything is on GitHub at github.com/net9876/ai-model-router-lab.

The Five Routing Strategies

The routing engine applies five strategies in priority order. The first one that produces a decision wins. If none of the first four match, there is a safe default.

Strategy 1: Environment

Rule: If the request comes from a development, test, staging, or CI environment — always use the cheap model.

This is the highest priority rule and the easiest win. There is no reason to pay premium model prices for developer tests, automated test suites, or staging environment requests. The quality of the answer does not matter — developers are just checking that the integration works.

_DEV_ENVIRONMENTS = {"dev", "development", "test", "testing", "local", "ci", "staging"}
def decide(self, prompt, cheap_model, premium_model, context):
    env = context.get("environment", os.getenv("APP_ENVIRONMENT", "production")).lower()
    if env in _DEV_ENVIRONMENTS:
        return ModelDecision(
            model=cheap_model,
            strategy=RoutingStrategy.ENVIRONMENT,
            reason=f"Non-production environment: {env}",
            confidence=1.0,
        )
    return None

In a typical company where developers call the API dozens of times per day for testing, this strategy alone can save 20-30% of total costs.

Strategy 2: Budget Cap

Rule: If the monthly spend has exceeded the configured budget — fall back to the cheap model for everything.

This is your financial safety net. You set a monthly budget (e.g., $20), and once it is reached, all traffic automatically shifts to the cheap model. No surprise invoices. No manual intervention.

def decide(self, prompt, cheap_model, premium_model, context):
    if context.get("override_budget"):
        return None  # caller explicitly bypassed this check
    spent = self.tracker.get_monthly_spend()
    if spent >= self.cap:
        return ModelDecision(
            model=cheap_model,
            strategy=RoutingStrategy.BUDGET,
            reason=f"Monthly budget cap reached (${spent:.4f} / ${self.cap:.2f})",
            confidence=1.0,
        )
    return None

Notice the override_budget flag in the context. If a critical production incident comes in at the end of the month when the budget is exhausted, you can explicitly bypass this check for that specific call.

Strategy 3: Task Type

Rule: Detect what kind of task the request is asking for and route based on a known classification.

Some tasks are always a good fit for cheaper models. Some tasks always benefit from premium models. Rather than scoring every request on a sliding scale, this strategy matches the request against a list of known task types and assigns the model with high confidence.

Premium tasks (always use GPT-4o):
– Security audits and vulnerability analysis
– Compliance assessments (GDPR, HIPAA, SOC2)
– Architectural design and system design
– Production incident analysis
– Comprehensive code reviews
– Deep data analysis

Cheap tasks (GPT-4o-mini is excellent):
– Summarization
– FAQ answers
– Draft emails and messages
– Simple translations and rephrasing

The detection is pure keyword matching — no extra API call, no added latency:

_TASK_KEYWORDS = {
    "summarize":        TaskType.SUMMARIZE,      # -> cheap
    "code review":      TaskType.CODE_REVIEW,    # -> premium
    "security":         TaskType.SECURITY_AUDIT, # -> premium
    "compliance":       TaskType.COMPLIANCE,     # -> premium
    "gdpr":             TaskType.COMPLIANCE,     # -> premium
    "architecture":     TaskType.ARCHITECTURE,   # -> premium
    "production bug":   TaskType.PRODUCTION_BUG, # -> premium
    "incident":         TaskType.PRODUCTION_BUG, # -> premium
    # ... and more
}

Strategy 4: Complexity Scoring

Rule: For requests where the task type is ambiguous, score the complexity of the prompt on a scale from 0.0 to 1.0 and route accordingly.

This is the most flexible strategy. It combines multiple signals into a single score:

• Token estimate: long prompts signal complex requests
• Premium keywords: words like “analyze”, “compare”, “architecture”, “optimize”, “comprehensive”
• Cheap keywords: words like “summarize”, “quick”, “simple”, “example”, “draft”
• Code block detection: code in the prompt usually means a technical review is needed
• Sentence count: many sentences often indicate a complex multi-part question

If the score is at or above the threshold (default 0.5), the premium model is selected. Below the threshold, the cheap model handles it.

def classify(prompt: str) -> ClassificationResult:
    score = 0.0
    # Token estimate
    estimated_tokens = max(1, len(prompt) // 4)
    if estimated_tokens > 2000:
        score += 0.4
    elif estimated_tokens > 800:
        score += 0.2
    # Code block detection
    if re.search(r"```|def |class |import |SELECT ", prompt):
        score += 0.15
    # Premium signal keywords
    for kw in _PREMIUM_SIGNALS:
        if kw in prompt.lower():
            score += 0.12
    # Cheap signal keywords (reduce score)
    for kw in _CHEAP_SIGNALS:
        if kw in prompt.lower():
            score -= 0.08
    return ClassificationResult(complexity_score=max(0.0, min(1.0, score)), ...)

Strategy 5: Default

Rule: If none of the above strategies produced a decision, route to the cheap model.

When in doubt, go cheap. The default is conservative by design — if we cannot confidently identify a reason to use the premium model, we assume the cheap model will do the job. This acts as a safety net and ensures no request ever falls through without an answer.

Azure Setup

The lab uses two Azure OpenAI deployments in the same resource. Setting them up is a single script:

az login
chmod +x setup/provision_azure.sh
./setup/provision_azure.sh

The script creates a resource group, provisions an Azure OpenAI service, deploys both models, and writes a .env file with your endpoint and API key automatically.

One important note from the lab: the GlobalStandard SKU is required for these model versions in Azure. The older Standard SKU will return a deprecation error. The script in the repository already uses GlobalStandard.

The resulting Azure resources:

Resource Group:  rg-ai-router-lab
  Azure OpenAI:  aoai-router-lab-XXXX
    Deployment:  gpt-4o-mini  (GlobalStandard, 100K TPM)
    Deployment:  gpt-4o       (GlobalStandard, 40K TPM)

When you are done with the lab, the teardown script removes everything:

./setup/teardown_azure.sh

Real Results from the Lab

These are not simulated numbers. This is the actual output from running five real scenarios against Azure OpenAI:

Total with routing: $0.0115
Total if all-premium: $0.0289
Savings: 60.1%

The security audit correctly went to GPT-4o — that is the whole point. For that request, you genuinely need the best model. For everything else, the cheaper model handled the job.

Simulated at scale

Running the router against a simulated batch of 50 mixed production requests — 30 simple, 20 complex — gives these numbers:

• 76% of requests routed to gpt-4o-mini
• 24% of requests routed to gpt-4o
• 66.2% total cost reduction vs always-using-gpt-4o

Extrapolated to 10,000 requests per month, the savings are meaningful for any growing application.

How to Add This to Your Existing Project

If you already have a project using Azure OpenAI and want to add the router, the change is minimal. Here is what you probably have now:

# Current code — hardcoded model
from openai import AzureOpenAI
client = AzureOpenAI(
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    api_key=os.environ["AZURE_OPENAI_API_KEY"],
    api_version="2024-02-01",
)
response = client.chat.completions.create(
    model="gpt-4o",   # <-- always premium, always expensive
    messages=[{"role": "user", "content": user_prompt}]
)
answer = response.choices[0].message.content

Option A: Direct Integration

Copy src/router/ and src/utils/ into your project and change three lines:

# New code — dynamic model selection
from router.router import build_router_from_env
from utils.azure_client import complete
router = build_router_from_env()
# This replaces your hardcoded model name
decision = router.route(user_prompt, context={"environment": "production"})
result = complete(
    deployment_name=decision.model.deployment_name,
    prompt=user_prompt,
)
answer = result.content
# Optional: inspect the routing decision
print(f"Used: {decision.model.deployment_name} | Reason: {decision.reason}")

Your application logic does not change at all. The router just gives you back the right model name for each request.

Option B: API Service

For larger projects or when multiple applications share the same AI infrastructure, run the router as a standalone API service:

uvicorn src.api.main:app --host 0.0.0.0 --port 8000

Then your existing application calls it like any other API:

import requests
response = requests.post("http://router-service:8000/chat", json={
    "prompt": user_prompt,
    "environment": "production",
})
data = response.json()
answer = data["content"]
# Full routing transparency in the response
print(f"Model: {data['routing']['model_deployment']}")
print(f"Cost:  ${data['actual_cost_usd']:.6f}")
print(f"Why:   {data['routing']['reason']}")

This option is better for microservices architectures where you want a single routing policy enforced consistently across all services.

The Cost Tracking Dashboard

Every request is logged to a local SQLite database with the model used, strategy applied, token counts, actual cost, and what it would have cost if you had used the premium model. The /stats endpoint aggregates this into a live savings report:

curl http://localhost:8000/stats

{
  "total_calls": 150,
  "cheap_calls": 114,
  "premium_calls": 36,
  "cheap_pct": 76.0,
  "total_cost_usd": 0.0312,
  "baseline_cost_usd": 0.1874,
  "savings_usd": 0.1562,
  "savings_pct": 83.4,
  "strategy_breakdown": [
    {"strategy": "task_type",   "calls": 72, "cost_usd": 0.0289},
    {"strategy": "complexity",  "calls": 48, "cost_usd": 0.0018},
    {"strategy": "environment", "calls": 24, "cost_usd": 0.0005},
    {"strategy": "default",     "calls": 6,  "cost_usd": 0.0001}
  ]
}

This is not just a vanity metric. It tells you which strategies are firing, what proportion of your traffic each one handles, and gives you the data to tune the routing thresholds over time.

Extending the Router

The routing engine is designed to be extended. Every strategy is a Python class that implements a single method:

class BaseStrategy(ABC):
    @abstractmethod
    def decide(self, prompt, cheap_model, premium_model, context) -> Optional[ModelDecision]:
        """Return a ModelDecision, or None to pass to the next strategy."""

You can add your own strategies for any business logic:

class UserTierStrategy(BaseStrategy):
    """Route based on the user's subscription tier."""
    def decide(self, prompt, cheap_model, premium_model, context):
        user_tier = context.get("user_tier", "free")
        if user_tier == "enterprise":
            return ModelDecision(
                model=premium_model,
                strategy=RoutingStrategy.DEFAULT,
                reason="Enterprise user — premium model always available",
                confidence=1.0,
            )
        return None  # pass to next strategy

Then add it to the strategy list in RouterEngine.__init__:

self.strategies = [
    EnvironmentStrategy(),
    BudgetStrategy(tracker, monthly_budget_usd),
    UserTierStrategy(),          # <-- your new strategy
    TaskTypeStrategy(),
    ComplexityStrategy(complexity_threshold),
    DefaultStrategy(),
]

Other ideas for custom strategies:
– Time-of-day routing: use cheap model during peak hours to control costs
– User quota routing: each user gets N premium calls per day, then falls back
– Experiment routing: A/B test cheap vs premium on a percentage of traffic
– Language routing: route non-English requests differently if models perform unevenly

When Routing Is Not the Right Approach

Model routing is powerful but it is not always the right tool. A few situations where you should think carefully:

When consistency matters more than cost. If your application needs deterministic, reproducible outputs — for compliance logging or audit trails — mixing two models can introduce subtle behavioral differences that are hard to trace.

When your requests are almost always complex. If 90% of your traffic is genuinely complex — large code reviews, multi-document analysis, long reasoning chains — the routing overhead is not worth the small savings. Just use the premium model with good prompt engineering.

When latency is extremely sensitive. The routing engine itself adds microseconds, not milliseconds — but if you route to the cheap model and it gives a lower-quality answer that requires a retry, your total latency increases. For hard real-time systems, test carefully.

When the task type boundary is unclear. If your domain makes it genuinely difficult to classify requests (highly specialized medical, legal, or scientific queries), the heuristic classifier may not route reliably without custom tuning.

What I Learned Building This

A few things surprised me during the lab:

The environment strategy is the biggest win, not the complexity strategy. I expected the complexity scoring to do most of the work. In practice, routing dev/test traffic to the cheap model is the single highest-value change for most teams — and it requires zero analysis of the prompt content.

GPT-4o-mini is better than I expected on moderately complex tasks. The architecture review scenario in the demo routed to gpt-4o-mini based on complexity scoring, and the answer was genuinely good. The model has improved significantly and handles many tasks that would have required GPT-4 a year ago.

The cost tracker changes how you think about AI. Once every request has a price tag attached to it in real time, you start thinking about prompt design and request batching very differently. The /stats endpoint is not just reporting — it is a feedback loop.

Keyword matching holds up better than expected. I was initially going to add a classifier model call to detect task type — which would have added cost and latency. The keyword matching approach is surprisingly effective for the most impactful routing decisions (security audits, compliance, production incidents) because those tasks use specific, predictable language.

Try It Yourself

The full lab is available on GitHub:

github.com/net9876/ai-model-router-lab

You can run the first two demo scripts without any Azure account or API key — they simulate routing decisions and calculate cost projections locally. The live demo requires an Azure subscription with Azure OpenAI access, which takes about five minutes to provision using the included script.

git clone https://github.com/net9876/ai-model-router-lab.git
cd ai-model-router-lab
pip install -r requirements.txt
# No API key needed for these
python demos/demo_basic.py
python demos/demo_cost_comparison.py

The repository includes everything: the routing engine, the FastAPI service, the cost tracker, unit tests, the Azure CLI provisioning script, and this write-up in the diagrams folder.

Conclusion

Model routing is one of those ideas that feels obvious in hindsight. Of course not every request needs your most capable — and most expensive — model. The question is just whether it is worth building the infrastructure to make the right choice automatically.

Based on the lab results, it clearly is. A 60-80% reduction in AI API costs with no change to application quality is a significant outcome, and the implementation is not complicated — a few hundred lines of Python, two Azure OpenAI deployments, and a handful of routing rules.

The approach also scales well as the AI landscape evolves. When new models arrive (and they will), you just add them to the configuration and update the pricing table. The routing logic does not change.

If you are running any LLM workload in production — or planning one — I would start here before spending time on more complex optimization strategies. The biggest cost savings are usually in the simplest decisions.

The full source code, Azure CLI scripts, and demo files are available at github.com/net9876/ai-model-router-lab. Questions and pull requests are welcome.