In the last twelve months, every major AI provider has had incidents, changed prices and shipped new models, sometimes in the same week. If your product calls one provider's SDK directly from twenty places in the codebase, each of those events becomes an engineering project.
An LLM gateway turns them into configuration changes.
What a gateway does
| Capability | Why you want it |
|---|---|
| One API for many models | Swap or add providers without touching feature code |
| Fallbacks and retries | Survive provider outages and rate limits |
| Model routing | Send each task to the cheapest model that passes your evals |
| Budgets and rate limits | Per tenant, per feature, per key, and no surprise bills |
| Caching | Exact and semantic caching for repeated requests |
| Logging and tracing | Prompts, responses, tokens, cost and latency in one place |
| Key management | Provider keys never live in application services |
| Guardrails | PII redaction and content filters at a single choke point |
Routing: where the savings are
Most AI features are a mix of tasks with very different difficulty:
- Classify, extract, tag, route: small, fast models
- Summarise, draft, rewrite: mid-tier models
- Multi-step reasoning, complex code, agent planning: frontier models
Teams that route by task commonly cut model spend by half or more without hurting quality. You need evals per task to prove it; see RAG vs fine-tuning vs long context for the eval approach. Our LLM cost optimisation guide covers the other levers.
Routing strategies, from simple to advanced:
- Static routing: each feature is configured with a model. Start here
- Fallback chains: primary model, then an alternative provider, then a smaller model with a degraded experience
- Cascade: try the small model first and escalate if confidence or validation fails
- Learned routing: a classifier picks the model per request. Only worth it at high volume
The options
Open source, self-hosted
- LiteLLM: the most widely used open-source proxy, with an OpenAI-compatible API, budgets, fallbacks and logging. Good default if you want control
Managed gateways
- Portkey, Helicone and similar: hosted gateways with strong observability and prompt management
- OpenRouter: one key and one bill for hundreds of models. Great for experimentation and breadth
- Cloud and platform gateways: Vercel AI Gateway, Cloudflare AI Gateway, AWS Bedrock and Azure AI Foundry. Convenient if you're already on that platform
Questions to ask
- Where is data processed and logged? (GDPR and customer contracts)
- Can you bring your own provider keys and contracts?
- Does it support streaming, tool calling and structured outputs for every model you use?
- How are budgets enforced: hard stop or alert?
Implementation checklist
- Route all model calls through one internal client that talks to the gateway
- Tag every request with tenant, feature and user for cost attribution, which also powers AI pricing
- Define fallbacks per feature, and test them by blocking the primary provider in staging
- Set budgets per tenant with alerts at 80%
- Redact PII before logging, and set retention periods
- Version prompts and model choices, and run evals before switching
Where it fits in an agent stack
In an agent architecture, the gateway sits between the agent loop and the models, while a durable workflow engine handles long-running state. Together they make agents reliable and affordable.
Bottom line
A gateway is a day or two of work that pays back the first time a provider has an outage, or the first time a cheaper model passes your evals. Do it before you have twenty call sites, not after.
Our AI integration team sets up gateways, routing and cost dashboards as part of production AI sprints. Get in touch.
Frequently asked questions
What is an LLM gateway?
A proxy layer between your application and model providers. It gives you one API for many models, plus fallbacks, retries, rate limits, budgets, caching, logging and key management in one place.
Does an LLM gateway add latency?
A well-run gateway typically adds only a few milliseconds, which is negligible next to model response times of hundreds of milliseconds or more. Caching often makes total latency lower.
What is model routing?
Sending each request to the most suitable model for the task: a small, fast model for classification or extraction, a frontier model for complex reasoning, and a fallback provider when the primary is degraded.