What Is An LLM Gateway?

An LLM gateway is a routing layer that sits between your application and one or more model providers. Your code sends requests to the gateway instead of calling OpenAI, Anthropic, Google, or another provider directly. The gateway handles provider selection, credential management, request translation, and the operational policies — caching, rate limits, retries, failover — that you do not want duplicated in every service.

The point is consolidation. A team running five LLM features against three providers without a gateway ends up with fifteen integration paths, fifteen places where API keys live, and no single place to enforce spend limits or approved models. A gateway turns that into one entry point with one credential store and one policy surface. That is why most enterprise deployments eventually build or buy one, whether they call it an LLM gateway, an AI gateway, or an internal orchestration service.

This is infrastructure, not intelligence. The gateway does not improve the model’s reasoning. It controls how requests reach the model and what happens when something goes wrong on the way there. For a fit-based comparison of six widely deployed products — LiteLLM, Portkey, OpenRouter, TrueFoundry, Kong AI Gateway, and Cloudflare AI Gateway — see our LLM gateway comparison guide.

Try Arize AX

Build better agents with Arize

Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.

Get started View AX docs

Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.

Key takeaways

• An LLM gateway is a routing and policy layer, not a model and not an observability platform. It decides which provider gets the request and under what constraints.
• The main value is operational: one integration surface, centralized credentials, provider failover, caching, rate limits, and cost attribution across teams.
• Most gateways are deployed as a proxy — change the API base URL and route traffic through the gateway — or as an SDK wrapper that normalizes provider APIs behind a single interface.
• The gateway is a hop in your trace. If it strips propagation headers or logs only aggregate metrics, you lose visibility into provider latency, token usage, and which model version actually served the request.
• A gateway is a single point of failure and a concentrated security surface. Treat its credential store, egress rules, and availability the same way you would any critical middleware.

What a gateway actually does

Provider routing. The gateway accepts a request with a model identifier and forwards it to the correct provider, often translating between API formats so your application code does not need a separate client for each one. Routing can be static — always send gpt-* to OpenAI — or dynamic, based on cost, latency, availability, or team policy.

Credential management. Provider API keys live in the gateway’s vault rather than in every service’s environment. Applications authenticate to the gateway; the gateway authenticates to providers. That separation is the main governance win: revoke a team’s access without rotating provider keys across the fleet.

Reliability policies. Retries with backoff, automatic failover to a secondary provider when the primary returns errors or times out, circuit breakers, and request timeouts. These policies belong in one place. When they are copy-pasted into every service, they drift.

Cost and usage controls. Per-team quotas, rate limits, budget caps, and spend attribution by project or feature. A gateway sees every request, which makes it the natural place to enforce limits before the bill arrives.

Caching and deduplication. Identical requests — same model, same prompt, same parameters — can be served from cache instead of hitting the provider again. The savings are real for eval runs, repeated system prompts, and batch workloads.

Request logging. Most gateways log token counts, latency, model identifiers, and status codes at minimum. Some also capture prompts and completions. Logging at the gateway is automatic when you use the proxy pattern, which is why gateways and observability are often discussed together — but logging is not the same as full tracing.

Proxy versus SDK integration

Two integration patterns dominate, and the choice affects how much you see in your traces.

Proxy (base URL swap). Point your existing OpenAI or Anthropic client at the gateway’s endpoint instead of the provider’s. No SDK change beyond the URL and an API key header. Every request passes through the gateway transparently. This is the fastest path to centralized routing and logging, and it is how most teams start.

SDK wrapper (unified interface). Use a library such as LiteLLM or a provider-agnostic SDK that speaks one API format and delegates to the gateway or directly to providers. The gateway may still sit behind the SDK as server-side infrastructure. This pattern gives more control over request shaping but requires adopting the wrapper in application code.

Both patterns put the gateway on the critical path. If it goes down, every LLM feature in the organization stops, not just one service. That tradeoff is worth making deliberately.

How this differs from neighboring concepts

Model provider. OpenAI, Anthropic, Google, and others host the models. The gateway calls them. It does not replace them.

General API gateway. Kong, Envoy, and similar tools route HTTP traffic, enforce auth, and terminate TLS. An LLM gateway adds provider-specific concerns: model identifier resolution, token counting, prompt/completion logging, provider API translation, and semantic caching. Some teams run both — a general API gateway at the edge and an LLM gateway behind it — but they solve different problems.

Observability platform. A gateway may log requests, but AI observability requires traces that capture the full execution path — retrieval, tool calls, agent steps, eval scores — not just the model call. Gateways that emit OpenTelemetry spans or integrate with tracing backends like Arize or Phoenix extend their value; gateways that only produce cost dashboards do not replace instrumentation in your application.

Agent framework. LangChain, LangGraph, and similar tools orchestrate agent loops, tool calling, and memory. A gateway routes the model calls those frameworks make. The framework decides what to do; the gateway decides how the request reaches the provider.

Prompt management system. A gateway may store prompts or route by prompt version, but prompt versioning as a discipline — immutable versions, labels, eval scores tied to version IDs — is a separate concern. A gateway without version tracking still needs a registry somewhere else.

Gateways and tracing

Every model call that passes through a gateway should appear as a span in the trace for that request. The gateway is a network hop, and distributed tracing only works if context propagates through it.

Three things to verify:

• Context propagation. The gateway forwards W3C traceparent headers (or your tracing vendor’s equivalent) to the provider and back. If it strips headers, provider-side latency disappears from the trace and attribution breaks at the gateway boundary.
• Span attributes. Model identifier, provider, token counts, latency, status, and the gateway’s own request ID should land on the span — not just in a separate cost dashboard.
• Gateway request ID correlation. The gateway assigns its own identifier to each request. That ID should appear on the span so you can join gateway logs to application traces when debugging a routing or failover issue.

For multi-service AI systems, the gateway is often the first place teams get automatic instrumentation — change the base URL and logging appears. That is a reasonable starting point, but it covers only the model call. Retrieval, tool execution, and agent orchestration still need instrumentation in the application to produce a trace worth debugging.

Where it goes wrong

• Treating gateway logs as observability. Token counts and latency at the gateway tell you the model call was slow. They do not tell you the retriever returned the wrong document or the agent called the wrong tool. Gateway metrics are one layer; they are not the trace.
• Single point of failure without a fallback path. If the gateway is down and there is no bypass to call a provider directly, every LLM feature stops. Some teams keep a direct-provider fallback for emergencies; others accept the dependency and invest in gateway availability.
• Credential concentration. Centralizing keys in a gateway vault is safer than scattering them across fifty repos — until the vault is compromised. Restrict egress, audit access, and rotate keys on a schedule.
• Policy drift across teams. A gateway only enforces what you configure. Teams that bypass it with direct provider keys undo the governance. Make the gateway the only approved path and detect bypass in network policy.
• Caching stale responses. Semantic or exact-match caching saves money until a prompt change or model update makes cached answers wrong. Cache invalidation policy belongs in the gateway configuration, not as an afterthought.

FAQ

Is an LLM gateway the same as an AI gateway?

Usually yes. “LLM gateway” emphasizes large language model routing; “AI gateway” is broader and may include image, embedding, and speech models. In practice the products overlap heavily, and the useful distinction is what the specific product routes, not the label.

Do I need a gateway if I only use one provider?

You can defer it, but the operational problems arrive quickly: API keys in every service, no per-team spend limits, no failover if the provider has an outage, and no single place to enforce approved models. A gateway pays off earlier than most teams expect, often before the second provider or the third team.

Can a gateway replace my observability platform?

No. A gateway sees model calls. It does not see retrieval quality, tool selection, agent planning, eval scores, or human feedback. Use gateway logging as one input to observability, not as a substitute for tracing the full agent execution path.

How does a gateway relate to prompt management?

Some gateways store prompts or support prompt templates, which overlaps with prompt management. The distinction is versioning and eval linkage: a prompt management system tracks immutable versions, ties them to eval scores, and supports label-based deployment. A gateway that stores a prompt string without version IDs gives you storage, not governance. Many teams use both — a gateway for routing and a registry for prompt lifecycle.

Should eval runs go through the gateway?

Yes, for the same reasons production traffic does: consistent routing, cost attribution, and the ability to compare eval spend across teams. One caution: caching can make eval runs non-reproducible if identical prompts hit cache instead of the provider. Disable caching for eval workloads or include a cache-busting parameter.