TL;DR
- An AI gateway only sees the traffic that chose to use it.
- Applications calling model providers directly do not appear in your gateway metrics, which means your logging, guardrails, cost attribution, and model policy silently cover a subset of usage while presenting as complete.
- Bypass is detectable across four layers: network egress, credentials, provider-side logs, and code.
- The fastest findings come from reconciling provider-held API keys against gateway-held keys and comparing provider billing against gateway-metered usage.
- The most durable fix is credential control, not network filtering, because a revoked key fails everywhere.
Why Applications Bypass AI Gateways
Almost all gateway bypass is convenience rather than intent. Four causes account for most of it.
The gateway arrived after the code. Services written before the gateway existed still call providers directly. They work, and the migration ticket has been in the backlog since the gateway launched.
Direct is the default in every SDK tutorial. Instantiate a client, read a key from an environment variable, make a call. That is the documented path in every quickstart. Pointing a base URL at an internal gateway is something a developer has to already know to do, and nothing in the provider’s documentation prompts them.
Local development leaks into production. Someone experiments on a laptop with a personal key. The code works, ships with the pattern intact, and the key follows it into environment configuration.
The gateway added friction once. Latency, a rate limit at the wrong moment, an unsupported model, or an outage. A developer routes around it during an incident, the incident ends, and the workaround becomes permanent because nothing broke.
None of these require carelessness. They require the direct path to be easier than the governed path, which it usually is.
The part that makes this dangerous
The failure is invisible in your own metrics.
Your gateway dashboard looks healthy. Request volume, latency, token spend, model distribution, policy violations, all reported cleanly. Every one of those numbers describes the traffic that came through the gateway. There is no metric for the traffic that did not.
That is a different problem from having no visibility at all. An organisation with no gateway knows it. An organisation with an unenforced gateway believes it has full visibility and makes decisions on that basis. Compliance attestations, cost forecasts, and model policy audits all rest on a number that measures a subset while presenting as a total.
This is the same structural problem described in our guide to shadow AI agents, applied at the API layer rather than the endpoint.
What Happens When an Application Bypasses the AI Gateway?
Worth being concrete, because you may need to justify this work to someone who thinks the gateway is already solving it.
Prompt and response logging. No record of what was sent or returned, removing the audit trail and the debugging path at once. Usually the first thing a compliance reviewer asks for.
Guardrails and redaction. Any PII stripping, content filtering, or policy enforcement configured at the gateway simply does not apply. If your data protection story depends on gateway-level redaction, bypass is a direct hole in it. The same applies to any input screening you rely on for prompt injection mitigation.
Cost attribution. Spend lands on the provider invoice with no internal owner. You know the total. You do not know which team, product, or service generated it.
Rate limiting and quota control. No protection against a retry loop or a runaway agent consuming a month of budget over a weekend.
Model policy. Approved-model lists, region pinning for data residency, and version control are unenforced. A service can call a model your architecture review rejected, in a region your legal team excluded. This matters more in multi-provider environments, where the approved set is larger and harder to police.
The compounding point: these are precisely the controls a compliance review will ask about, and “we have a gateway” is not an answer if using it is optional. Our overview of AI agent governance covers how these controls fit into a wider framework, and European enterprise AI governance covers the regulatory dimension.
How to Detect AI Gateway Bypass at the Network Layer
The control most teams reach for first, and the one most commonly overestimated.
Egress allowlisting. Enforcement. Block outbound traffic to provider endpoints from everything except the gateway’s egress address or proxy. It works, and the maintenance cost is easy to underestimate. Provider address ranges change, and when they do, address-based rules fail silently in one of two directions: legitimate traffic breaks, or a new range becomes an open path nobody notices. Domain-based egress policy at a proxy ages considerably better. Source current ranges from the provider, not from any article including this one.
DNS control. Partial enforcement. Route internal DNS resolution for provider domains to an internal resolver or sinkhole, so name resolution forces traffic through the gateway. Cheaper to maintain than address rules and easier to audit, because the policy lives in one place. The limit is that it controls resolution, not connectivity. Anything using a hardcoded address or an alternate resolver is unaffected.
Service mesh policy. Enforcement. Inside a cluster, mTLS and egress policy restrict which workloads may reach external endpoints at all, forcing model traffic through the gateway service. The strongest network-layer control available, because it is identity-based rather than address-based: the policy attaches to the workload, not to a destination that can change. Applies only where you already run a mesh.
Flow log analysis. Detection only. VPC flow logs and firewall logs surface outbound connections to provider endpoints that did not originate from the gateway. This changes nothing about what is allowed, but it needs no policy change to run, which makes it the fastest thing to do on day one.
The limit worth naming
Network controls do not cover what leaves the network.
A developer laptop off VPN. A personal device. A contractor’s machine. A cloud function in an account predating your landing zone. None sit behind your egress policy, and no firewall configuration reaches them.
That is the boundary of the approach rather than a flaw in it, and it is why the layers below matter more than they first appear.
How to Prevent AI Gateway Bypass With Credential Controls
The most durable control, and the one to do first.
The principle is straightforward: a call cannot bypass the gateway if the application does not hold a credential that works without it. Network filtering makes bypass difficult within a perimeter. Credential control makes it impossible everywhere, regardless of where the code runs or whose device it runs on.
Remove raw provider keys from applications. Audit environment variables, secret stores, CI configuration, container images, and infrastructure-as-code for provider keys held by anything other than the gateway. In most organisations this single audit surfaces the majority of bypass paths, because the key is the bypass path. It also reduces the data exposure risk of keys sitting in places they were never meant to persist.
Issue gateway-scoped credentials instead. The gateway holds provider keys; applications receive internal credentials valid only against the gateway. A direct provider call then fails on authentication rather than being blocked at the network layer, which is a cleaner failure and far easier to diagnose. The developer gets a clear error instead of a timeout.
Scope credentials per service. One credential per service rather than a shared key across an environment. This makes usage attributable and revocation surgical. It is the same principle applied to agents in agent governance: shared credentials destroy attribution, and attribution is what makes every other control enforceable.
Reconcile against the provider’s key inventory. Most providers expose the keys issued under an organisation account. Comparing that against what your gateway holds takes minutes and is one of the highest-signal checks available. Any key the gateway does not hold is a bypass path or an orphan, and both need action. What each provider exposes varies by vendor and plan tier, so check your account.
Sequencing recommendation
Do credentials before network.
Most teams start with egress filtering, because it feels like the security control and deploys without touching application code. Then they spend months maintaining address lists and chasing false positives.
Credential rotation is more work up front and dramatically less work permanently. It also has the property network filtering lacks: it works on the laptop, the contractor’s machine, and the forgotten cloud account.
How to Detect AI Gateway Bypass Using Provider Logs and Billing
Underused, and available without touching your own infrastructure at all.
Organisation usage logs. Detection. Provider dashboards expose usage at varying granularity. Requests from addresses that are not your gateway are bypass by definition, and requests attributed to keys the gateway does not hold are the same finding from another angle. Granularity differs by provider and plan tier, so establish what your account exposes before building a process on it.
Billing reconciliation. Detection. Compare the provider invoice against gateway-metered usage. A persistent delta is bypass. This is a coarse signal, but it has two useful properties: it requires no engineering work, and finance teams notice it. In organisations without a security-led programme, this is often what surfaces the problem in the first place.
Request signature patterns. Detection. Traffic from your gateway carries a consistent user agent and header signature. Direct SDK calls carry the SDK default. Where a provider exposes this, it separates the two cleanly without any correlation work.
None of these are enforcement. All three tell you whether you have a problem, and the first two size it, which is usually what you need before you can get budget for the rest.
How to Detect AI Gateway Bypass in Code and Configuration
The cheapest place to catch bypass is before it ships.
Static analysis for direct SDK instantiation. Enforcement, at build time. Scan for provider SDK clients constructed without a base URL pointing at the gateway. This is a simple, stable pattern and it belongs in continuous integration rather than in a quarterly audit.
A pattern to adapt, not a production rule:
# Flag provider client instantiation without a gateway base_url
# Syntax will differ by SAST tool; adapt the matcher to your stack.
pattern: $CLIENT = $PROVIDER.Client(...)
pattern-not: $CLIENT = $PROVIDER.Client(..., base_url=$GATEWAY_URL, ...)
message: Provider client created without gateway base URL.
severity: ERROR
This catches the common case. It misses clients constructed dynamically, wrapped in a factory, or configured through a settings object several layers away, so treat it as a floor rather than a ceiling.
Secret scanning for provider keys. Detection, escalating to enforcement. Standard secret scanning already recognises provider key formats. The change worth making is to policy rather than tooling: in a gateway-enforced environment, no application should hold a provider key, so a provider key in a repository is an incident rather than a warning. Fail the build.
Dependency and configuration review. Detection. Some libraries call providers directly by default through their own configuration path, so a dependency added for an unrelated reason can introduce a bypass nobody wrote. Review what your dependencies do at the network layer, particularly anything in the agent framework and orchestration space, where direct provider calls are common.
A flow log query pattern, to adapt to your logging stack:
# Outbound connections to model provider endpoints
# NOT originating from the gateway's address.
filter dstport in [443]
and dstaddr in $PROVIDER_ENDPOINT_SET
and srcaddr != $GATEWAY_EGRESS_ADDR
| stats count(*) by srcaddr, dstaddr
| sort count desc
Populate $PROVIDER_ENDPOINT_SET from current provider documentation and refresh it on a schedule. A stale set produces false confidence, which is worse than no query.
How to Detect and Fix AI Gateway Bypass: A 6-Step Process
Six steps, ordered to produce findings fastest rather than to build the strongest control first.
- Reconcile provider-held keys against gateway-held keys. Minutes to run, no infrastructure change, highest signal available.
- Compare provider billing against gateway-metered usage. Confirms whether a gap exists and roughly how large it is.
- Query flow logs for provider traffic not originating from the gateway. Locates which workloads are responsible.
- Scan repositories for direct SDK instantiation and provider keys. Finds the code behind the traffic.
- Fix credentials. Revoke, reissue through the gateway, redeploy. Most bypass stops here permanently.
- Add egress and CI enforcement to prevent recurrence. Policy last, once you know what enforcing it would break.
Steps one and two are the ones most teams skip, and they are the fastest route to knowing whether you have a problem at all. Both can be done in an afternoon by one person with the right access.
If this is part of a broader programme, our playbook on taking agents to production covers the wider sequence. Platform teams and CTOs inheriting this work will recognise most of the pattern from earlier shadow IT efforts, and the same friction problems described in enterprise AI agent challenges apply here.
What an AI Gateway Does and Does not Govern
An AI gateway controls which model calls happen, on whose credentials, and what policy applies to the prompt and the response. That is a real and necessary control, and nothing in this article is solved by an agent platform.
What the gateway answers: which model was called, whose key was used, what was sent, what came back, what it cost, and whether the content violated policy.
What it does not answer: which agent made the call and why, what tools it invoked afterwards, whether the action it took was approved, and who is accountable for the outcome. A gateway sees one HTTP request. It does not see the reasoning loop around it, the tool call that followed, or the decision that request was part of.
Where the layers meet: both need the same thing, which is attribution. A gateway credential per service and an agent registry entry per agent are the same control applied at different layers. Neither works if the identity underneath is shared.
To be explicit: Lyzr is not an AI gateway and does not replace one. It sits above whichever gateway you run. The Control Plane provides agent registration and ownership, per-agent permission scoping, human approval gates on consequential actions, and decision traces that answer what an agent did and on what basis. Responsible AI and hallucination management controls apply at the behavioural layer, where the gateway has no visibility.
If you are evaluating tooling for the observability half of this problem, our comparison of AI agent observability platforms covers the landscape, including where gateways and agent-level tracing overlap.
Ready to close the gap between model access and agent accountability? See how the OpenController handles agent registration, permission scoping, and decision traces, read the Lyzr documentation, or book a demo to walk through it with our team.
Frequently asked questions
What is an AI gateway?
A proxy sitting between your applications and model providers, handling routing, credential management, rate limiting, prompt and response logging, and policy enforcement for model calls.
What is the difference between an AI gateway and an API gateway?
An API gateway manages traffic to your own services. An AI gateway manages outbound calls to model providers and adds model-specific concerns like token accounting, prompt logging, model routing, and content policy.
How do I know if applications are bypassing my AI gateway?
Start by reconciling provider-issued API keys against the keys your gateway holds, then compare provider billing against gateway-metered usage. Both take minutes and both are high signal.
Can egress filtering stop AI gateway bypass?
Partly. It works inside your network perimeter and breaks when provider address ranges change. Credential control is more durable, because a revoked key fails everywhere including on unmanaged devices.
Why do applications bypass the AI gateway?
Usually convenience rather than intent: code written before the gateway existed, SDK defaults that call providers directly, local development keys that shipped, or a workaround adopted during a gateway outage.
What do you lose when a call bypasses the gateway?
Prompt and response logging, guardrails and redaction, cost attribution, rate limiting, and approved-model policy. Effectively everything the gateway was bought for.
Is an AI gateway a security control or an observability tool?
Both, but only when enforced. Without enforcement it observes the traffic that opted in, which makes its dashboards misleading rather than merely incomplete.
How do I enforce AI gateway usage in CI?
Add static analysis for provider SDK clients instantiated without a gateway base URL, and treat provider API keys found in repositories as build failures rather than warnings.
Does an AI gateway govern AI agents?
No. It governs model calls. Agent behaviour, tool use, permissions, and approvals sit above it and need separate governance.
What about calls from laptops or unmanaged devices?
Network controls do not reach them, which is why credential control matters more. For unsanctioned tooling running locally, see our guide to detecting AI coding agents on endpoints.
Book A Demo: Click Here
Join our Slack: Click Here
Link to our GitHub: Click Here
