TL;DR
- Microsoft Foundry vs Lyzr for multi-agent evaluation comes down to scoring versus deciding: Foundry scores and traces agents, and Lyzr Opencontroller acts on those scores.
- Foundry’s built-in system evaluators judge the orchestrator or final agent, so scoring each handoff means writing custom LLM-as-judge evaluators.
- Foundry’s model leaderboard ranks models on public benchmarks. Benchmarking your own system needs your own golden set.
- Foundry now traces LangChain, LangGraph and other OpenTelemetry frameworks, so the real differences are where each one runs and what a failing score triggers.
- Opencontroller adds release gates, gateway policy and auto-hardening, and runs in your VPC, on-prem or air-gapped.
- Most Azure teams will run both.
Picture a three-agent claims workflow on Microsoft Foundry: an intake agent, a policy agent and a settlement agent (a composite scenario). On the test set, it passes Task Completion on nearly every case. But two weeks into production, an adjuster spots settlements above the customer’s coverage limit. The policy agent had been passing an outdated limit to the settlement agent, which did the maths correctly on the wrong number. Every final answer looked complete, so the evaluator had nothing to flag.

That gap, between a good final answer and a bad handoff, is the real question behind Microsoft Foundry vs Lyzr for multi-agent evaluation. Here’s what Foundry (formerly Azure AI Foundry) measures out of the box, what you build yourself, and where Lyzr fits.
The short answer: Foundry gives you built-in agent evaluators, OpenTelemetry tracing and model leaderboards inside Azure. Lyzr Opencontroller adds release gates, runtime policy, simulation and auto-hardening across frameworks and clouds. If your agents live in Azure, use Foundry to score and Opencontroller to decide what happens next.
Why final-answer scores hide multi-agent failures
Multi-agent systems are spreading faster than the ways to test them. Gartner reports a 1,445% surge in multi-agent system inquiries from Q1 2024 to Q2 2025. The research on how they fail is less cheerful. UC Berkeley researchers built MAST, the first taxonomy of multi-agent failures, from traces across seven popular frameworks. They found 14 failure modes in three groups: system design issues, inter-agent misalignment and task verification. They also found that the performance gains of multi-agent systems on popular benchmarks are often minimal.
Inter-agent misalignment is exactly the claims workflow’s problem, and it rarely shows in the final answer. So a multi-agent system needs scoring at four levels: the end-to-end outcome, each agent’s own work, each handoff, and the cost and latency each agent adds. Skip the handoff level and you’re judging a relay race by the clock without checking who dropped the baton.
Multi-agent evaluation on Microsoft Foundry: what’s built in
Foundry’s agent evaluators come in two families. System evaluators judge the outcome: Task Completion, Task Adherence, Task Navigation Efficiency, Intent Resolution and Customer Satisfaction. Process evaluators judge each tool call: Tool Call Accuracy, Tool Selection, Tool Input Accuracy, Tool Output Utilization and Tool Call Success. The built-in evaluator list still marks Task Completion, Task Adherence, Intent Resolution and Customer Satisfaction as preview.
The line that matters for multi-agent work: Microsoft says system evaluators apply to single agents and, in multi-agent systems, to the main orchestrator or the final agent responsible for the task. Nothing built in scores the policy agent’s handoff to the settlement agent.
You write that part yourself. Microsoft’s training module on multi-agent evaluation frameworks teaches success metrics for coordination quality and handoff effectiveness, calibrated LLM-as-judge patterns for multi-agent chains, and multi-judge consensus across models to reduce bias. It works, but it’s code your team owns, calibrates and maintains.

Tracing is the stronger story. Foundry captures inputs, outputs, tool usage, retries, latency and cost in Application Insights. Tracing is generally available for prompt and hosted agents, but workflow and external agents are still in preview. In June 2026, Microsoft extended tracing and evaluation to agents built on LangChain, LangGraph, the OpenAI SDK, Microsoft Agent Framework and any custom framework. After launch, continuous evaluation and the Agent Monitoring Dashboard track quality trends.
There are two catches though. Task Navigation Efficiency needs ground truth: the expected tool sequence for each case. And the Foundry (classic) SDK docs say Tool Call Accuracy supports a listed set of tools, such as File Search, Azure AI Search, Code Interpreter and OpenAPI. Call anything else and the evaluator returns a pass, with a note that it can’t evaluate that tool, unless you wrap it as a user-defined tool.
What Foundry’s model leaderboard can’t benchmark
Search “Foundry benchmarking” and you land on the model leaderboard. It ranks models on quality, safety, estimated cost and throughput, and compares up to three side by side. That’s useful for shortlisting a model for each agent role. Microsoft’s own troubleshooting note admits the limit: public benchmarks use standardized datasets and might not reflect performance on your data.
Benchmarking a multi-agent system means a golden set of fixed scenarios with expected outcomes and expected handoffs, scored on every version, per agent and per handoff. If version 1.4 drops on settlement accuracy, you need to know whether the settlement agent got worse or the data reaching it did.
My view: teams spend too long on leaderboards because reading one is easy. Only the golden set predicts production.
Your golden set says what passed. Can you show what’s running in production, and what it’s worth? Read the white paper:
Microsoft Foundry vs Lyzr for multi-agent evaluation, side by side
Up until here, we’ve looked at Foundry on its own: it scores agents, traces what they do and benchmarks models. Lyzr Opencontroller also evaluates agents, but its main job starts where Foundry’s ends. It’s a control plane that sits above your agents, wherever they’re built or run, Foundry included, and acts on the scores: whether a version ships, what policy applies in production and how a failing agent gets hardened. Here’s how the two compare.
| Microsoft Foundry | Lyzr Opencontroller | |
|---|---|---|
| Main job | Build, evaluate and observe agents | Find, ship, run and improve agents built anywhere |
| Evaluation | System and process evaluators; several in preview | Continuous evaluation, simulation and auto-hardening |
| Model benchmarking | Leaderboard: quality, safety, cost, throughput | Not a model leaderboard |
| Frameworks | Foundry agents plus OpenTelemetry-instrumented frameworks | LangGraph, CrewAI, Strands, custom |
| When a score fails | Scores, traces and dashboards | Evaluations, approvals and release controls at the deployment handoff |
| After launch | Continuous evaluation and monitoring | Guardrails, budgets and policy in its gateways |
| Where it runs | Azure | SaaS, VPC, on-prem, air-gapped |
The top rows favour Foundry: in our view, its evaluator catalogue is broader. The lower rows are where the two split: what happens when a score fails, what governs an agent after launch, and where it can run. Opencontroller can run alongside Foundry or instead of it. For Azure teams, alongside makes sense, and the claims workflow shows why.
Where Foundry evaluation stops
Say your team writes the handoff judge, and it catches the stale coverage limit on a Thursday. What happens on Friday?
In Foundry, whatever your process says. An evaluation run is a report. A low score doesn’t hold the release on its own; that call lives in a CI step someone can skip or a meeting someone can miss. Monitoring flags drift after launch, but flagging isn’t stopping.
It gets worse as the system grows. The settlement agent may come from another team on another framework, or never get a test set. Across twenty agents, nobody can say which golden set version tested which agent version. Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value or inadequate risk controls. A red dashboard that changes nothing is a risk control that isn’t working.
What’s missing isn’t another evaluator. It’s a layer that knows every agent, ties scores to versions and acts on them.
How Lyzr Opencontroller makes multi-agent scores decide
Opencontroller is that layer, a control plane built around four stages: Find, Ship, Run and Improve. Agents stay where they already run, and each stage closes one gap from the previous section.
Find: Opencontroller discovers and governs agents, models, tools, data and workflows across the estate, regardless of where agents were built or where they run. The settlement agent from the other team gets an owner, and that owner gets a test set.
Ship: Per its architecture, Opencontroller applies evaluations, approvals and release controls around the deployment handoff, while your CI keeps building, scanning and signing. A failing handoff score can hold that version, so nobody has to remember the Friday meeting.
Run: Guardrails, budgets and policy sit in Opencontroller’s gateways, an agent gateway and an LLM gateway in its architecture. When something your golden set missed reaches production, the response can be a pause, an approval or a rollback, not just an alert.
Improve: Lyzr pairs continuous evaluation and simulation with auto-hardening, with equal depth for LangGraph, CrewAI, Strands and custom agents. Production evidence lands in the same system of record as releases and approvals, so the stale-limit failure becomes a golden-set case instead of a postmortem slide.
It runs as SaaS or in your VPC, on-prem or air-gapped, and customer-managed deployments don’t require control-plane telemetry, credentials or application evidence to be exported to Lyzr. For Azure teams, Lyzr runs inside your Azure VNet, natively connected to Azure OpenAI, Foundry and Fabric. Keep Foundry’s evaluators and traces. Opencontroller makes their results count.
Under Opencontroller, the claims workflow’s bad handoff would have held a release instead of reaching an adjuster’s desk.
Book a demo to see Opencontroller turn your Foundry evaluation results into release gates.
Frequently asked questions
They do different jobs. Foundry offers a broad built-in evaluator catalogue and Azure-native tracing. Lyzr Opencontroller turns results into release gates, runtime policy and auto-hardening across frameworks and clouds. Azure teams often run both.
System evaluators score the orchestrator or final agent, and process evaluators score tool calls. Per-agent and per-handoff scoring needs custom LLM-as-judge evaluators built with the Foundry SDK.
No. It compares models on public datasets for quality, safety, cost and throughput. To benchmark a multi-agent system, score every version against your own golden set.
Yes. Since June 2026, Foundry traces and evaluates agents built on LangChain, LangGraph, the OpenAI SDK, Microsoft Agent Framework and other OpenTelemetry-instrumented frameworks.
Book A Demo: Click Here
Join our Slack: Click Here
Link to our GitHub: Click Here
Got a use case in mind?
8 weeks from use case to agents running in production.
Platform, people and FDEs, all in. Bring your environment. We'll co-build and stay until it's live.
Talk to UsFree resource
101 AI Agents Use Cases
100+ production-ready AI agent use cases across Marketing, Sales, HR, Banking, Insurance, Operations and a lot more
Download now