TL;DR
- The choice: in the LLM gateway vs direct provider SDKs decision, direct SDKs win on speed to ship, native features and the shortest path.
- Where a gateway wins: once several providers, teams or agents share production, an LLM gateway puts failover, keys, budgets and policy in one place.
- Start direct: an SDK suits a prototype or single-provider feature. Add a gateway when a second provider, shared spend or an outage arrives.
- Check passthrough: a unified API can quietly drop provider-native features like prompt caching.
- Mind the gaps: a gateway only governs the calls that go through it. Find the ones that don’t.
Your first LLM feature took a weekend. You installed the provider’s SDK, put a key in an environment variable and shipped. Eighteen months later there are three SDKs across nine services, a retry wrapper someone wrote during an incident, and provider keys in four secret stores. In August, OpenAI shut down the Assistants API that two of your agents used. Then finance asks what each team spends on models, and nobody can say.
That’s usually when the gateway question comes up. Here’s how to answer it with the costs on both sides in view.
What you’re actually choosing between
A direct provider SDK is a vendor’s own client library, such as OpenAI’s, Anthropic’s or Google’s GenAI SDK. Your code uses it to call that one vendor’s API. An LLM gateway is a service between your apps and every provider: apps call one endpoint, usually OpenAI-compatible, and the gateway handles routing, keys, limits and logs.
Many teams sit in a third place without ever choosing it: a homegrown wrapper or a multi-provider client library. It gives you one interface in code, but keys, budgets and logs still live in each service.
Gateways come in three shapes: self-hosted, managed, or part of a control plane in your own cloud. The shape decides where your keys and prompts travel.
The team in our story runs SDKs plus a wrapper, so they’re choosing for real.
LLM gateway vs direct provider SDKs: the trade-offs side by side
Direct SDKs cost less up front and keep every provider feature. A gateway costs more to run and pays you back in portability, resilience and control.
| Dimension | Direct provider SDKs | LLM gateway |
| Time to first call | Minutes: install, add a key, call | Hours to days: deploy or subscribe, then configure providers |
| Switching models or providers | A code change in every service; each SDK has its own request, response and error shapes | Change a model name or a route; app code mostly unchanged |
| Provider-native features | Full access the day they ship (prompt caching, reasoning output, new tools) | Depends on the gateway: a unified schema can drop them unless native formats pass through |
| Outages and rate limits | Retries hit the same provider; cross-provider fallback is code you write | Fallback to another provider or region by configuration, if the fallback is tested |
| Cost visibility | Per-provider dashboards; spend by team is do-it-yourself | Spend by team, app or key in one place; budgets enforced before the call |
| Keys and security | Raw keys spread across services; policy set per service | Keys in one place; one point to enforce PII checks, injection checks and model allowlists |
| Observability | Instrument each SDK yourself | One log and trace stream for every call |
| Latency | Shortest path | One extra hop; small next to model time, but it adds up in agent loops |
| Operational burden | Lowest | You run, scale and patch it, or trust whoever does |
| Failure blast radius | One provider’s bad day hits the features that use it | The gateway becomes a single point of failure and a high-value target |
Neither column is free. The next two sections price each side.
What direct provider SDKs get right, and what they quietly cost
Direct SDKs are the right call when one team ships on one provider. There’s nothing extra to run, and you get new features the day they ship. The costs arrive later and spread across teams, which is why they’re easy to miss.
- SDK churn, paid once per service. OpenAI removed the Assistants API on August 26, 2026, and the reusable prompt objects its migration guide pointed to are scheduled to shut down on November 30, 2026. Google’s legacy Gemini SDKs were deprecated on November 30, 2025. Every service on those SDKs ran its own migration.
- Keys everywhere. Each service holds raw provider keys, so one rotation becomes a change in every codebase.
- Retries aren’t failover. An SDK retry calls the same provider again. During an outage, that adds load without adding a way out.
- Telemetry by hand. OpenTelemetry’s GenAI conventions are still marked as in development and moved to their own repository in June 2026. Instrumenting each SDK yourself means tracking a moving spec in every codebase.
For our team, the Assistants shutdown touched four services owned by three teams. Two migrated on time. One shipped a broken streaming handler.
Direct integration is cheap per service and expensive per organization.
What an LLM gateway fixes, and the three costs it adds
A gateway gives you one endpoint, one place for keys and budgets, and failover by configuration. In return, you add a component to every request path, and possibly an API that can’t express everything your providers can.
- The lowest-common-denominator API. A unified schema can only carry what it knows how to translate. Anthropic’s own docs say its OpenAI-compatible layer doesn’t support prompt caching, doesn’t return Claude’s thinking, ignores strict on tool schemas and silently ignores most unsupported fields. Anthropic says the layer “is not considered a long-term or production-ready solution for most use cases.” Ask any gateway whether it passes native request formats through or only translates them.
- A single point of failure that holds every key. If the gateway goes down, every provider goes down with it. If it’s compromised, every key it holds is exposed. That risk became real for one popular open-source gateway twice in 2026.
- Failover is only as good as the fallback. A backup model that formats output differently can break a parser without raising an error. Providers also stumble together: on September 3, 2026, ChatGPT, Claude and Grok all hit outages the same morning. Test every fallback route against your evals.
The team’s first gateway trial wiped out the prompt caching savings on one service, because the translation layer dropped the cache parameters. Nobody noticed for a week.
How to measure gateway latency honestly
Overhead is the gateway path minus the direct path, at the same load, against the same upstream. Measure it with your real features switched on, such as logging, guardrails and routing, not plain forwarding. Run the gateway in the same region and network as your apps, since the extra network hop often costs more than the gateway’s own code. Then multiply by the number of model calls in one agent task.
When to use a direct SDK and when to use an LLM gateway
Use direct SDKs while one team ships on one provider and an outage is survivable. Move to a gateway when a second provider, a shared budget, regulated data or an uptime promise to customers enters the picture.
| Your situation | Choose | Why |
| Prototype or internal tool on one provider | Direct SDK | Speed matters most, and there’s nothing to share yet |
| One latency-critical path, such as voice | Direct SDK, recorded as an exception | Every hop counts; keep it visible and reviewed |
| A feature that depends on one provider’s native capability | Gateway with native passthrough, or a direct exception | Don’t lose the feature to translation |
| Two or more providers, or plans to switch | Gateway | Switching becomes configuration, not code |
| Several teams sharing one model budget | Gateway | Spend by team, with limits enforced |
| Regulated data or a strict security review | Gateway inside your trust boundary | One place for policy, and keys stay home |
| Many agents calling models, tools and each other | Gateway plus a control plane | A model gateway alone doesn’t see tool or agent calls |
Most teams end up with a hybrid, which is fine when it’s deliberate. Make the gateway the default and allow direct calls only as named, reviewed exceptions, or the hybrid slides back into SDK sprawl. Ready to shortlist? Compare LLM gateways feature by feature.
How to move from direct SDKs to a gateway without breaking production
Treat the move as an inventory problem first and a code change second. Changing the base URL moves the call. It doesn’t move keys, retries, budgets or the provider features each service relies on.
- Inventory every call site. Search repos and configs for SDK imports, provider hostnames and key names.
- List the native features each call uses, such as caching, structured outputs and reasoning. They decide passthrough versus translation.
- Move keys into one secret store, then rotate the old ones.
- Route in log-only mode first, and switch on enforcement and fallback once evals pass.
- Block direct provider traffic from production networks, with named exceptions.
The team’s inventory found 14 call sites. They knew about 11.
A gateway only governs the calls that go through it
A gateway controls the model calls you route through it. It can’t see calls that skip it, and it wasn’t built for the rest of what agents do: calling tools and calling other agents.
Three months after the move, a scan turned up two services still calling an SDK directly. One was built after the migration. Meanwhile, the team’s agents were calling MCP tools and handing work to other agents, and none of that passed through the model gateway.
So the question changes. It’s no longer “gateway or SDK?” It’s “is every call, from every agent, governed?” The gateway decision is about models. The governance decision covers everything your agents touch.
How Lyzr Opencontroller gives you a gateway without lock-in or blind spots
By now the team needs two things: gateway benefits without adopting another SDK, and a way to see and govern what a model gateway can’t. Lyzr Opencontroller is a control plane for AI agents with an LLM gateway built in. In its own words, “A model gateway covers one block of ten.”
| What the team needs | What Lyzr Opencontroller provides |
| Failover, budgets and keys across providers | LLM gateway: “Virtual keys, team budgets, rate limits, routing, fallback and caching across model providers.” |
| Keys out of every service | Your secrets manager “stays in place,” and customer-managed deployments “don’t require control-plane data, credentials or telemetry to reach Lyzr.” |
| Calls that skip the gateway | Find: “Read-only scans across clouds, clusters and developer laptops surface shadow agents and build one live inventory.” |
| Tool calls and agent-to-agent calls | Agent gateway: “Every caller, human, app or agent, is authenticated, authorized and rate-limited.” Guardrails “screen every input, output and tool call.” |
| Fallback routes tested before release | Ship: “Multi-turn, red-team and regression evals gate every release, and the evidence ships with the version.” |
| The gateway inside your boundary | “Deploy in your VPC, on-prem or air-gapped.” |
Adoption starts small: “Change an endpoint, not your application.” Opencontroller works through standard interfaces, with OpenAI-compatible APIs for models, MCP for tools, A2A for agents and OpenTelemetry for traces. That way, governance doesn’t become the next lock-in.
For the team, keys move back to their own secrets manager. The two services still calling SDKs directly show up in the inventory with owners. Fallback routes pass release evals before they’re switched on, and every trace carries cost and owner into the observability tools they already run.
Book a demo to see Opencontroller on your own agents.
Where to start this week
Count your call sites: every SDK import, provider hostname and key, by team. For each one, write down the provider feature it can’t lose. Then pick your default and your exceptions, in writing. That turns the LLM gateway vs direct provider SDKs question from a debate into a policy.
Frequently asked questions
A direct provider SDK is a vendor’s client library that sends requests straight to that vendor’s API. An LLM gateway sits between your apps and many providers, exposing one endpoint and handling routing, failover, keys, budgets and logging in one place.
For prototypes, single-provider features owned by one team, and latency-critical paths where every hop matters. It’s also the surest way to use a new provider feature on day one. Once you run a gateway, record each direct path as a reviewed exception.
It depends on where it runs and which features are on. Measure the gateway path minus the direct path at the same load. For one call it’s usually small next to model time, but it multiplies across the model calls in an agent task.
It can move lock-in rather than remove it. A gateway with its own SDK or config format becomes the new dependency. Prefer one with an OpenAI-compatible endpoint that passes provider-native formats through, so leaving means changing an endpoint, not your code.
You can, if the gateway only translates to one common API. Anthropic’s OpenAI-compatible layer, for example, doesn’t support prompt caching or return Claude’s thinking. Check whether your gateway passes native requests through, and test the features you rely on.
Book A Demo: Click Here
Join our Slack: Click Here
Link to our GitHub: Click Here
Got a use case in mind?
8 weeks from use case to agents running in production.
Platform, people and FDEs, all in. Bring your environment. We'll co-build and stay until it's live.
Talk to UsFree resource
101 AI Agents Use Cases
100+ production-ready AI agent use cases across Marketing, Sales, HR, Banking, Insurance, Operations and a lot more
Download now
