Lexis+ AI and Westlaw AI-Assisted Research built their pitch on retrieval-augmented generation solving hallucination. A Stanford RegLab study tested that pitch directly: Magesh et al. found the two tools, along with Ask Practical Law AI, “each hallucinate between 17% and 33% of the time,” on queries where each answer cited real sources. Retrieval worked. Nobody scored whether generation used what retrieval found.
That gap is what RAG evaluation exists to close, and this guide compares the tools built for it, from a pip-installable metrics library to a platform that turns a failing score into an enforced policy.
Get the short answer
- Need a failing score to trigger a policy, not just a dashboard note? Pick Lyzr. Agent Eval scores the pipeline; Opencontroller decides what happens next.
- Want the fastest way to tell retrieval and generation apart? Pick Ragas. Context precision and recall score the retriever; faithfulness scores the generator, from one pip install.
- Need RAG checks inside your CI pipeline? Pick DeepEval.
deepeval test runfails a build the way a broken unit test would. - Want the RAG Triad with tracing built in? Pick TruLens. Groundedness, context relevance and answer relevance, MIT-licensed, now with native MLflow scoring.
- Running retrieval against vector databases, tool calls or MCP servers? Pick Arize Phoenix. Its retrieval evaluator scores any of those sources with an explanation attached.
- Already live on LangChain or LangGraph? Pick LangSmith. RAG evaluators sit next to the trace you’re already debugging.
What is RAG evaluation?
RAG evaluation is the practice of scoring a retrieval-augmented generation system in two separate parts: whether the retriever fetched the right context for a query, and whether the generator’s answer stays faithful to that context instead of drifting into invention. The two scores point to different fixes. A low retrieval score means the chunking, ranking or knowledge base needs work; a low faithfulness score with strong retrieval points to the model or the prompt.
Four metrics carry most of the weight here. Context precision and context recall grade the retriever. Faithfulness, sometimes called groundedness, and answer relevance grade the generator. TruLens groups three of these into what it calls the RAG Triad: context relevance, groundedness, and answer relevance, scored without a labeled reference answer. RAG evaluation is one slice of the broader AI agent evaluation tools set.
Retrieval problem or generation problem? Here’s how to tell
A wrong answer from a production agentic RAG pipeline is one symptom with two possible causes, and teams waste real time fixing the wrong one. Before reaching for any tool, it helps to know which half of the pipeline to blame.
| What you’re seeing | Likely cause | What to check |
|---|---|---|
| The answer cites the wrong document, or misses an obvious one | Retrieval | Context precision, context recall |
| The right documents came back, but the answer invents details not in them | Generation | Faithfulness, or groundedness |
| The answer is accurate but doesn’t address what was actually asked | Generation | Answer relevance |
| Retrieval looks clean, yet the model still ignores the context it was given | Prompt or model choice | Faithfulness, despite a high context precision score |
Most teams only run this check once, before launch, and rarely again. LangChain’s State of Agent Engineering survey found 52.4% of organizations ran offline evaluations on a fixed test set, but only 37.3% scored live production traffic (see offline versus online evaluation). That gap holds until a knowledge base gets re-indexed or a vendor swaps the embedding model, quietly moving an answer from accurate to hallucinated. That’s the broader measurement gap between AI pilots and production.
That’s the blind spot the Stanford study caught in production: a system can pass every offline check and still hallucinate on a third of live queries, because the test set never reflected what users actually ask. Lyzr’s free Hallucination Risk Assessment is built around that exact gap and takes about five minutes to run against your own pipeline.
Six RAG evaluation tools reviewed, with strengths and trade-offs
1. Lyzr: evaluation inside a governed agent platform
Every tool below can tell you which half of that table a RAG answer landed in. None can stop the agent from sending it anyway. Lyzr’s Agent Eval scores factual accuracy, context relevance and logical groundedness through HybridRAG Evaluation, which checks relevance across public and internal knowledge bases in one pass. Opencontroller then sits in the request path, so a failing score can pause the action, route it for approval, or roll it back instead of just appearing in a log somewhere. It works the same across AWS, Azure, and GCP deployments. Lyzr isn’t a free, standalone metrics library like the next five; we note below where Ragas, DeepEval and TruLens go deeper on raw scoring.
| Capability | Evaluation libraries provide this | Lyzr adds this |
|---|---|---|
| Score the retriever and generator | Yes, through separate precision/recall and faithfulness metrics | HybridRAG Evaluation across public and internal sources in one score |
| Trace the reasoning | Some, through spans and annotations | Reasoning Trace follows agent logic end to end, verifying sources used |
| Catch a bad answer before release | Via a CI gate, if one is wired up | Opencontroller evaluates before production, not just in a pipeline step |
| Act on a failing score | No, it’s logged or flagged | Pauses, flags for approval, or rolls back the action in the request path |
Pick it if you are a platform or compliance lead whose RAG system feeds agents that act on the answer, especially where an audit trail is non-negotiable.
Skip it if you are a developer still iterating on a chunking strategy for a prototype. Start with Ragas or DeepEval, and come back once the pipeline feeds something that acts on its own answers.
2. Ragas: the library that keeps retrieval and generation separate
Ragas, introduced in a 2023 paper by Es et al., remains the fastest route to a split retrieval-versus-generation score: context precision and recall grade what came back from the retriever, faithfulness and response relevancy grade what the model did with it. Most of its scoring is reference-free, which matters since most production traffic lacks a labeled answer. Testset generation builds evaluation data from your own documents, and Tool Call Accuracy extends the same logic to agents. Its GitHub home has since moved from the original Exploding Gradients organization to Vibrant Labs; the library and its Apache 2.0 license are unaffected.
License: Apache 2.0, pip-installable.
Best for: teams that need to know, fast, whether a bad answer came from retrieval or generation.
Not ideal if: you need a dashboard; Ragas returns a dataframe, not a UI.
3. DeepEval: RAG checks that live inside pytest
DeepEval, built by Confident AI, treats a RAG regression the way it would treat a broken unit test: deepeval test run wires contextual precision, recall and relevancy, plus faithfulness and answer relevancy, into a CI pipeline that fails the build on a score drop. Contextual precision and contextual recall need a labeled expected output; faithfulness and answer relevancy don’t, so scoring can start before labels exist. Its Synthesizer generates golden test cases from source documents, solving the usual excuse for skipping evaluation.
License: Apache 2.0, self-host or Confident AI’s hosted cloud layer.
Best for: engineering teams that want a RAG regression gate before every merge.
Not ideal if: you need production tracing, which lives in the paid Confident AI tier.
4. TruLens: the RAG Triad, now talking to MLflow
TruLens groups three reference-free checks into the RAG Triad: context relevance (did retrieval pull the right material), groundedness (does the answer follow from it), and answer relevance (does the answer address the question asked). TruEra built it, Snowflake acquired TruEra in May 2024, and the library has kept shipping as MIT-licensed open source since. Its April 2026 release added native MLflow scoring through mlflow.genai.evaluate, so MLflow 3.10+ teams can pull RAG Triad scores into an experiment they already track.
License: MIT, self-hosted.
Best for: teams that want the RAG Triad, and MLflow users who’d rather not add a second eval surface.
Not ideal if: you need ranking metrics like precision@k without a curated ground-truth set.
5. Arize Phoenix: retrieval scoring wherever the context came from
Phoenix’s retrieval relevance evaluator doesn’t care whether context came from a vector database, a tool call, an MCP server, or a web search; it scores each item against the original request with an explanation attached. Faithfulness and correctness evaluators check generation the same way, logged on the trace. It runs on OpenInference over OpenTelemetry, which keeps it portable across observability backends (see Best Tools for LLM Tracing and Debugging). Arize’s managed sibling, Arize AX, is being folded into Dynatrace, which announced a roughly $915 million acquisition of Arize in August 2026.
License: Elastic License 2.0; self-hosting stays free regardless of the Dynatrace deal.
Best for: teams whose retrieval sources are more varied than a single vector store.
Not ideal if: you need CI/CD gating built in; that part lives in Arize AX, not the open Phoenix library.
6. LangSmith: RAG evaluators next to the trace you already use
LangSmith builds RAG evaluation on top of the trace viewer LangChain and LangGraph teams already have open: datasets assemble from curated cases or production traces, and retrieval relevance and groundedness evaluators run through its pytest and Vitest/Jest integrations, so a RAG regression test sits in the same suite as the rest of your tests. Multi-turn online evaluators extend the same scoring to full conversations, useful once a RAG system fields follow-ups against context retrieved earlier in the thread.
License: commercial, with a free developer tier; Cloud, Hybrid, or self-hosted on the Enterprise plan.
Best for: teams already living in LangChain or LangGraph who want RAG scores beside the trace, not in a separate tool.
Not ideal if: your stack sits outside that ecosystem; the retrieval evaluators work elsewhere but are only tutorial-documented.
Compare all six RAG evaluation tools side by side
| Tool | License | Scoring coverage | Acts on a failing score | Decision trail | CI/CD fit | Tracing or dashboard |
|---|---|---|---|---|---|---|
| Lyzr | Commercial | HybridRAG Evaluation: relevance across public databases and internal knowledge bases, plus truthfulness, context relevance, groundedness | ✅ Opencontroller: a pause, an approval, or a rollback before a bad output reaches a customer | ✅ Identity-linked: every decision traceable, every policy enforceable | 🟡 Opencontroller gates before production, not a CI-specific check | ✅ Agent Registry, versioning, and monitoring |
| Ragas | Apache 2.0 | Context precision, recall, faithfulness, response relevancy | ❌ None; returns a dataframe | ❌ None | ❌ Not built in | ❌ No dashboard or trace viewer |
| DeepEval | Apache 2.0 | Contextual precision, recall, relevancy, faithfulness, answer relevancy | 🟡 Fails the CI build via deepeval test run, not enforced on live traffic | ❌ None | ✅ Native via deepeval test run | 🟡 Production tracing lives in the paid Confident AI tier |
| TruLens | MIT | RAG Triad: context relevance, groundedness, answer relevance | ❌ None | ❌ None | ❌ Not built in | ✅ Built-in dashboard |
| Arize Phoenix | Elastic License 2.0 | Retrieval relevance (any source), faithfulness, correctness | ❌ None | ❌ None | 🟡 Not in Phoenix; lives in Arize AX | ✅ OpenInference over OpenTelemetry |
| LangSmith | Commercial | Retrieval relevance, groundedness, relevance, correctness | 🟡 Fails the pytest/Vitest suite, not enforced on live traffic | ❌ None | ✅ pytest, Vitest/Jest | ✅ Built on the LangChain/LangGraph trace viewer |
Which RAG evaluation tool fits your stack
| If your situation is… | Pick… | Why… |
|---|---|---|
| A RAG agent acts on its own answer and needs a failing score to stop it | Lyzr | Scoring plus enforcement in one request path |
| You need to know fast whether retrieval or generation is at fault | Ragas | Clean split, reference-free, one pip install |
| You want the check wired into your merge pipeline | DeepEval | Native pytest integration |
| You’ve standardized on the RAG Triad or already run MLflow | TruLens | MIT core, native MLflow scoring since version 2.7 |
| Your context comes from more than one kind of source | Arize Phoenix | Scores vector stores, tools, MCP and web search alike |
| You live in LangChain or LangGraph | LangSmith | Evaluators next to the trace you already debug |
Run these three tests before you commit
See also: Best Tools for AI Agent Simulation and Testing.
- The reference-free test. Feed it real production queries with no labeled answer. If the tool needs ground truth for every metric, it won’t scale past your first test set.
- The drift test. Re-index your knowledge base, or swap a document template, then re-run the same suite. A tool worth keeping should show the score move.
- The enforcement test. Force a faithfulness score below your threshold and watch what happens. “It gets logged” means an evaluation tool. “The action gets paused” means governance.
Scoring is step one, not the finish line
A RAG system that passes its evaluation suite still needs something watching the traffic that suite didn’t cover. That’s a governance maturity question: Lyzr’s Agent Governance Maturity Assessment is a short, free self-assessment built around exactly that gap.
See Opencontroller on your own RAG pipeline
Six tools above can tell you a RAG answer failed. One can also stop it from reaching a customer. If your evaluation suite is solid and the next question is what happens in production, a live demo will show the difference on your own stack. For the layers around this one: Open-Source LLM Observability Platforms, Best Tools for Real-Time AI Agent Monitoring, and Best Tools for AI Agent Session Logging.
Book a demo of Opencontroller →
FAQ
RAG evaluation scores a retrieval-augmented generation system in two parts: whether the retriever fetched the right context and whether the generator answered faithfully from it. Teams run it on test sets before release and on live traffic afterward, using metrics such as context precision and faithfulness.
Context precision grades the retriever: did the right chunks come back. Faithfulness grades the generator: do the answer’s claims trace back to those chunks. A system can score high on one and low on the other, with a different fix each way.
Not for every metric. Faithfulness, answer relevance and the RAG Triad run reference-free, so Ragas, TruLens and DeepEval’s answer relevancy work on live traffic with no labeled answer. Ranking metrics like precision@k and DeepEval’s contextual precision and recall still need one.
It reduces the odds, but offline testing alone missed the hallucination rates the Stanford RegLab study found in production. Offline testing plus online scoring plus a governance layer that can act on a failing score catches more than either alone.
For scoring quality, usually yes. Ragas, DeepEval, TruLens and Phoenix all document real metrics, but none document an audit trail tied to identity or a way to stop an action in progress, which compliance reviews ask for next. Not sure how many RAG pipelines you’re running? Start with the AI Agent Sprawl Audit.
Ragas or DeepEval. Both are free, pip-installable, and score without a labeled test set. Add TruLens or Phoenix once you need tracing on production traffic, and a governance layer once agents act on their own answers. For non-RAG calls, see Best Tools for LLM Evaluation.
Book A Demo: Click Here
Join our Slack: Click Here
Link to our GitHub: Click Here
