7 Best Platforms for LLM Inference and Serving 

TL;DR 

• Agents, not chatbots, now drive inference bills. Gartner expects cost per agentic workflow to rise more than fivefold through 2028, and recommends routing each step to the cheapest model that can handle it. 
• Pick a serving platform by where requests may run. Then put one control layer in front to hold the keys, route, fail over and cap spend. 
• Lyzr Opencontroller is that layer, deployable in your VPC, on-prem or air-gapped. 
• Amazon Bedrock and Microsoft Foundry suit cloud-committed teams. Together AI and Fireworks AI suit open-model traffic. Baseten runs managed serving in your VPC. vLLM with llm-d is for teams that want to own the engine. 

October was a bad month to have an inference key stored on someone else’s platform. On October 9, Fireworks AI told customers that misused internal credentials had been used to read environment variables in some evaluation jobs, third-party API keys included. It found no evidence that models or inference traffic were accessed. Within two weeks OpenCode and Railway disclosed intrusions too, and a poisoned LiteLLM release went hunting for provider credentials. The Cloud Security Alliance titled its write-up “Inference Providers as the Prize.”  

The same month, llm-d reached 1.0, and IBM reported that open models on it ran 24 hours of agent work for about $26, against roughly $300 on a frontier model. 

Side by side, they’re the brief for choosing platforms for LLM inference and serving: the engine matters, and so does whatever sits in front of it, holding the keys and picking which model answers each call. 

Why the bill grows while tokens get cheaper 

Gartner’s August 17 prediction: inference cost per agentic workflow will rise more than fivefold through 2028, because an agentic reasoning call costs a provider at least five times a basic chatbot exchange. Analyst Will Sommer warns that defaulting to generic autonomy brings “unbounded costs orders of magnitude higher” than an optimized setup. 

Gartner’s fix is inference tiering: each step goes to the cheapest model that can handle it. iStart’s write-up of the research says routing and workload optimization alone can cut costs by up to 60%. 

The catch: tiering needs one layer that sees every model, and most serving platforms only route inside their own catalog. 

Four questions that shrink the shortlist 

1. Where may a request run? Public API, your cloud account, your data center or an air-gapped network. This alone rules out half the market. 
2. Who holds the keys? CSA’s advice after October: a key per application, provider-side spend caps, and credentials injected from your own secrets manager, never pasted into dashboards. 
3. What happens on a 429? A replica helps; a different provider helps more. 
4. What does a finished task cost? Per-million-token prices are the rate card. Finance will ask about cost per resolved ticket. 

The Enterprise Agent Estate Is the Next Infrastructure Decision: Agents and the models they call now belong in the same budget conversation as your cloud. Here’s how to plan for it. Read the white paper → 

The 7 best platforms for LLM inference and serving 

1. Lyzr Opencontroller: the control layer in front of every endpoint 

Opencontroller runs no GPUs. It’s a control plane in the request path in front of models, tools and agents, and its LLM gateway decides where each call lands: virtual keys, team budgets, routing and fallback across providers, and per-agent rules on which models an agent may call. Locally hosted LLMs can sit behind it too. 

The cost lever is shadow routing. A task goes first to a small open-weight model. If the answer misses a quality bar you set, the gateway sends the original request to a frontier model, and the trace logs each model tried and its token cost. That’s Gartner’s inference tiering, run by the gateway. 

Your secrets manager stays in place, and credentials never need to reach Lyzr. It deploys in your VPC, on-prem or air-gapped, with guardrails for prompt injection, personal data and secrets. 

Pick it if your agents call several inference platforms, or keys and spend need one owner. Look elsewhere if you serve one model to one app. 

2. Amazon Bedrock: the AWS default, rebuilt on a new engine 

Bedrock’s 2026 story is Project Mantle, a distributed inference engine with OpenAI-compatible endpoints that AWS introduced in December 2025. AWS added 18 open-weight models that month and six more, served on Mantle, in February. A Projects API, added February 26, tags workloads so cost is tracked per project. Custom Model Import serves your own weights with no import fee, and batch on selected models costs half the on-demand rate. 

Pick it if AWS is home and Claude does the heavy lifting. Look elsewhere if a second cloud is coming. 

3. Microsoft Foundry: three deployment types, one endpoint 

Since Build 2026, Foundry pairs pay-per-token and provisioned throughput with Managed Compute: dedicated GPUs for open and custom-weight models, billed hourly per accelerator, with scale-to-zero. All three share one endpoint, the same SDKs, Entra ID and private networking. Fireworks-served open models joined in public preview in March. 

Pick it if your security team runs on Entra and you have Azure commitments. Look elsewhere if you need routing beyond Azure. 

4. Together AI: open models at serious volume 

Together raised $800 million at an $8.3 billion valuation in July, then signed a $240 million deal with IBM for a dedicated NVIDIA HGX B300 cluster on IBM Cloud, due in Q1 2027. It reported about 400 trillion tokens a month. Buy serverless per token, dedicated endpoints billed per minute, or GPU clusters. Switching serverless to dedicated is a model-parameter change; batch halves the price on selected models. Enterprise customers can also deploy it in their own VPC. 

Pick it if open models carry most of your traffic and you’d rather start pay-per-token. Look elsewhere if you want closed frontier models from the same vendor. 

5. Fireworks AI: fast serving for fine-tuned models 

Fireworks raised $1.505 billion at a $17.5 billion valuation in July, reporting $1 billion-plus in annualized revenue and over 40 trillion tokens a day. Over 95% of those tokens, it says, come from models specialized on customers’ data, so its buyers are teams that fine-tune open models. Options run from serverless to reserved and dedicated capacity. Bring Your Own Cluster, in private preview, runs the serving stack on Kubernetes you own. 

Pick it if latency is part of your product and the models are your own fine-tunes. Look elsewhere if inference must be pinned to a non-US region. 

6. Baseten: managed serving inside your VPC 

Baseten closed a $1.5 billion Series F in June at valuations up to $13 billion. Regulated buyers come for self-hosted mode: the workload plane runs in your AWS, GCP or OCI account, so inference payloads stay inside your network, while Baseten runs the control plane. Hybrid mode places each workload in your VPC or Baseten Cloud. 

Pick it if compliance wants your network boundary but not your own autoscalers. Look elsewhere if you need a fully air-gapped deployment. 

7. vLLM with llm-d: own the whole engine 

vLLM shipped v0.32.0 on October 18; its creators raised $150 million for Inferact in January to commercialize it. Two days later llm-d hit 1.0, a CNCF project founded with Red Hat, IBM, Google, CoreWeave and NVIDIA, adding multi-tenancy, non-disruptive upgrades and agent-aware serving that reuses cached context. On an agentic coding workload, Google reports prefix-aware routing plus host memory offload cut median time to first token by over 85% and lifted throughput 2.4 times versus a plain Kubernetes Service. 

Pick it if you have platform engineers and steady, high-volume traffic. Look elsewhere if nobody wants the 3 a.m. page. 

Seven platforms on one grid 

✓ built in · ◐ partial or preview · ✗ not offered or not the focus. Opencontroller runs no GPUs, but you deploy the control plane. Foundry routes only within Foundry. Editorial assessment, October 2026. 

Opencontroller leads on five of six columns and pairs with any of the others. Our view: most agent programs need that pairing by year two, as the Gartner math pushes them toward frontier, open and on-prem models together. 

What a routing policy looks like in practice 

Take an IT service-desk agent, wired through Opencontroller to three platforms from this list: 

1. Redact. Guardrails catch names and employee IDs at the gateway before any model sees the ticket. 

2. Try small. An open-weight model on vLLM in your data center drafts the reply; a classifier scores it. 

3. Escalate. Failed drafts go to a frontier model on Bedrock, unchanged. 

4. Fail over. A 429 from Bedrock reroutes the call to Foundry. 

5. Account. The team’s virtual key carries a monthly budget, and each trace shows what the ticket cost. 

Watch the escalation rate. If most tickets escalate, the small model is wrong for the job; if almost none do, your quality bar is too loose. Tune it and you’re cutting LLM spend without changing models. 

AI Agent Sprawl Audit Agents calling providers directly with their own keys never show up in a gateway. Find yours. Take the audit → 

Start with the platform, then decide who holds the keys 

Pick the serving platform your compliance team can live with. Then put one layer in front that owns routing, spend and credentials, so agents nobody registered can’t slip around it. Explore Opencontroller or book a demo to map your endpoints to a routing policy. 

FAQs 

Related reading: Open-source LLM observability platforms · Best tools for AI policy enforcement at scale · Best tools for real-time AI agent monitoring 

Book A Demo: Click Here
Join our Slack: Click Here
Link to our GitHub: Click Here