One Shot RAG: How Lyzr Rebuilt Multi-Knowledge-Base Retrieval

TL;DR

• Classic agentic RAG lets the model loop: retrieve, read, retrieve again,  3 to 25 LLM calls per question, each one adding seconds and tokens.
• OneShot RAG replaces the loop with a fixed two-call pipeline: a tiny planner that routes your question across every knowledge base in parallel, and the agent’s own answer call acting as the synthesizer.
• One planned retrieval pass. No loops.
• In our 11-knowledge-base benchmark it routed 99.3% of 201 probe queries to the right knowledge base, scored higher on answer accuracy than the agentic loop (1.43 vs 1.29 on a 2-point LLM-judge scale), and delivered first tokens faster (5.33s vs 5.96s median).

The hidden tax of letting your agent “figure it out”

The standard way to give an AI agent access to your documents is deceptively elegant: wrap your knowledge base in a tool, hand the tool to the agent, and let the ReAct loop do its thing. The agent decides when to search, reads what comes back, decides whether to search again, and eventually answers.

It works. It’s also expensive in exactly the ways that hurt production systems.

• Every “search again?” step is a full LLM call. The entire conversation, system prompt, and tool schemas go back to the model, which thinks, then maybe emits another search.
• The loop ran 3 to 25 iterations on knowledge-base questions in our production traces, driving the pathological p95 latencies (~3 minutes) that made us rebuild this feature.
• Retrievals are sequential. Search #2 can’t start until the model has read the results of search #1.
• Each hop paid connection overhead. The old implementation opened a fresh HTTP session per hop, paying DNS + TLS handshake (50–200ms) every single time.

The loop’s flexibility is real. But for the overwhelming majority of questions, “what’s our wire transfer approval limit?”, “compare the consumer and commercial policies”, the agent doesn’t need to iterate. It needs to search the right places, once, and answer.

That observation is the entire feature.

A: the agentic loop, climbing the same circular staircase, arms full, clock running. 

B: one shot , a single straight crossing, two marks total.

Why it’s called “one shot”

The name refers to the retrieval, and it’s worth being precise, because it’s easy to misstate (we’ve even caught our own release notes getting it wrong).

OneShot RAG makes exactly two LLM calls per turn, never more, no matter what goes wrong:

1. The planner: a small, cheap model (gpt-4o-mini by default, temperature 0) that does not answer the question. Its only job is to read a catalog of your knowledge bases and emit a JSON plan: which KBs to search, with what search strings, with what metadata filters.
   
2. The synthesizer: and here’s the trick: there is no dedicated synthesizer call. The retrieved chunks and answering rules are injected into the agent’s system prompt, and the agent’s ordinary response call, the one it was going to make anyway, becomes the synthesizer.

Between those two calls sits the “shot”: every planned sub-query fans out in parallel over a shared, keep-alive HTTP/2 connection pool, results are merged by score, and the best chunks land in the prompt. One planned pass. Retrieval never loops back through the model.

So the honest arithmetic against the agentic loop:

And the design guarantee that makes “one shot” more than a slogan: every fallback and retry lives in the retrieval layer, never the LLM layer. Planner returned garbage? Retrieve from everything. 

Filters matched nothing? Strip them and re-fan-out. Still nothing? One bounded retry. Worst case is still two LLM calls, the retries are HTTP calls, which cost milliseconds, not model calls, which cost seconds and money.

Stage by stage: what actually happens

1. The planner sees names, not IDs

Each knowledge base is registered with a semantic name, a description, optional filterable fields, and example queries. The planner’s catalog looks like:

An early version asked the planner to emit raw 24-character hex database IDs. Small models copy long hex strings poorly, and worse, they bias toward the first ID in the list. Switching to short semantic names fixed both problems at once: names are easy to copy verbatim, and the model routes on meaning (matching your question against the description) instead of position.

The plan comes back as strict JSON:

The planner is also the query rewriter: it strips filler, expands pronouns, and preserves identifiers (names, dates, amounts), which is why we deliberately don’t expose HyDE or other query-expansion modes downstream. The rewrite already happened.

2. Guardrails that assume the planner will be wrong

A 4o-mini call at temperature 0 is reliable, not infallible. Everything it emits is validated in plain code: unknown KB names are dropped, never guessed; filters are checked against each KB’s declared whitelist (a filter the planner invented gets deleted before it can silently exclude your documents); and if the plan is unusable, the system falls back to querying every KB with the raw question.

Two guards are worth calling out because they came from real failures:

The empty-only retry replaced an earlier, buggier version: retrying whenever the best retrieved score looked “too low” (< 0.25). That was wrong because cosine scores aren’t comparable across corpora of different density. A perfect hit in a diffuse 700-page textbook scores ~0.1; a mediocre hit in a one-page form scores ~0.6. The threshold made retrieve-all overrule the planner’s correct choice.

Every branch in that fallback cascade is an HTTP-layer decision. None of them re-invokes an LLM; that’s the “one shot” invariant holding under failure.

3. The merge: where multi-KB RAG actually breaks

Merging results from multiple knowledge bases sounds like a solved problem: use Reciprocal Rank Fusion, the textbook answer. We did. It caused our best bug story.

With 11 knowledge bases live, the system started answering every question, factoids, out-of-scope questions, even chitchat, from the same fixed set of six KBs. Wire-transfer questions returned transformer-paper chunks. The retrieval itself was healthy: querying the wire KB directly returned chunks scoring 0.64 while the junk that “won” scored 0.2–0.32. Routing was broken, not search.

Score-aware merge: eleven drawers, and the tallest card wins its slot no matter which drawer it came from, measured, not positional.

The root cause is subtle and general enough that if you’re building multi-KB RAG, check your merge right now: RRF scores by rank, and every list’s rank-0 chunk gets the identical score, 1/(k+0). When N knowledge bases each return their own top chunk, all N tie. A stable sort then breaks the tie by insertion order, so whichever KBs happened to be listed first always won, and KBs 7–11 starved at exactly 0%.

The fix leans on a property of our setup: every KB is embedded with the same model (text-embedding-3-large), so absolute cosine scores are comparable across KBs for ranking. The merge now:

• Ranks by absolute score instead of rank position.
• Keeps RRF only as a whisper-weight agreement bonus (0.1 × the RRF term, at most ~0.002, enough to break genuine ties and nothing more).
• Applies a relative floor at 55% of the top score to drop off-topic tails.
• Guarantees each deliberately-routed KB its top chunk, floor or no floor. If the planner chose two KBs for a comparison question, both sides of the comparison survive the merge, even when one corpus scores systematically lower.

(Notice the apparent contradiction with the retry guard above: scores are comparable enough to rank within one query, but not stable enough to serve as an absolute quality threshold across corpora. Both are true. Getting that distinction wrong cost us two separate bugs.)

4. The synthesizer that refused too much

The chunks get formatted into a citation-friendly block, grouped by source document, with [doc_id:chunk_id] markers deliberately truncated, since long OCR-generated filenames measurably degrade citation rates, and injected into the agent’s system prompt along with answering rules. Our first version of those rules was binary: answer from the chunks, or refuse.

The live benchmark punished us for it, producing a 55% refusal rate on questions where the right chunks were demonstrably present, versus just 1% for the agentic loop. The literature calls this over-refusal and attributes it to exactly that binary phrasing (RefusalBench, arXiv:2510.10390).

The rewrite made the response graduated:

The prompt even bans un-grounded hedging: no “typically”, no “in most cases” unless a chunk says so.

The numbers, honestly

Methodology: We ran this as an internal benchmark on a live dev environment, comparing OneShot RAG against the legacy agentic tool-loop on the same corpus. The test corpus spanned 11 knowledge bases, covering financial policies, ML textbooks, WebRTC networking, Python, internship reports, and more.

To check routing, we ran 201 probe queries and measured whether each one got sent to the correct knowledge base. To check answer quality, we used a set of 21 questions covering per-KB, cross-KB, out-of-corpus, and chitchat cases, with each answer graded by an LLM judge on a 0–2 scale. These are internal numbers, not an independent benchmark.

An earlier 28-query benchmark during development measured time-to-first-token at 1.98× faster (4.12s → 2.08s); the gap varies with corpus and load, but the direction is consistent: the user starts reading sooner.

And yes, the agentic loop wins one row, total wall-clock, and keeps an edge on genuinely multi-hop questions where iterative refinement (search → read → search about what you just read) is the point. Parallel decomposition is not iteration, and we’d rather tell you that than have you discover it. For the dominant single- and multi-KB lookup workload, OneShot answers more accurately, starts responding sooner, and costs a bounded, predictable two LLM calls instead of an open-ended loop.

What we’d tell anyone building this

• Route by name and description, never by ID. Small models copy hex badly and position-bias hard.
• Rank-based fusion breaks at rank zero. If your KBs share an embedding model, use the scores; keep RRF as a tiebreaker at most.
• Never threshold on absolute cosine across corpora. Density makes 0.1 in one KB better than 0.3 in another. Retry on empty, not on “low.”
• Refusal behavior is a first-class prompt-engineering surface. Binary “answer or refuse” instructions over-refuse; graduated instructions with an explicit high bar for refusal fixed a 55% → single-digits problem.
• Put every fallback below the LLM line. Retries that re-invoke the model silently turn your “fast path” back into the loop you were escaping.

Where this goes next

OneShot RAG still plans at query time: a small LLM call deciding, per question, where to look. The next step in this arc is already in progress at Lyzr: a knowledge engine that compiles knowledge bases ahead of time into typed, page-cited artifacts organized into task-shaped contexts, so the shape of the answer is decided before any question arrives, and even the planner call disappears from the hot path. One shot, and nothing left to plan.

It’s early, but we’re already measuring it on public ground, the same way we benchmarked OneShot: run it, publish what happened, including the part that broke.

We pointed the engine at EnterpriseRAG-Bench, Onyx’s open benchmark built from a synthetic company’s realistic documents, using the github source slice (8,052 real files) and its 39 questions, in the engine’s free compile mode (no embeddings, no LLM calls anywhere in the pipeline), scored on their judge-free document-recall metric:

That 69% is the zero-cost floor: pure keyword-and-salience retrieval with no semantic search at all, on documents it compiled for free. The semantic-embedding layer that lifts it, and the full judged benchmark at corpus scale, is the follow-up post.

If you’re on Lyzr Studio today: OneShot RAG is live, attach multiple knowledge bases to an agent, give each one a good name and description (they’re load-bearing: that’s what the planner routes on), and the platform does the rest.

Book A Demo: Click Here
Join our Slack: Click Here
Link to our GitHub: Click Here