Back to Overview
Case Study #2Enterprise Knowledge & Operations RAGArchitecture Modernization & Retrieval Engineering

Knowledge Assistants (RAG Chatbots)

From 15–30 minutes of manual searching to 5 seconds: grounded RAG assistants with 100% out-of-scope refusal.

Open WebUIbge-m3pgvectorLiteLLMvLLMDoclingPostgreSQL

Retrieving a technical answer drops from 15–30 min of manual search to <5 s with citations (measured); out-of-scope refusal improved from 62% to 100% on our evaluation set.

Step 4 · Grounded answer with citation

IMPACT SIMULATION · V-2Simulated example

Knowledge Assistant

Follow one question from search to a cited answer, or watch it decline when the question is off-topic.

  1. Ask
  2. Search
  3. Rank
  4. Answer
How do I fix error E-104 in the teller app?
Document chunks
FAQManualsProceduresPoliciesError codesGuides
5 chunks matched
Top 5 chunksScore
01Operations Guide v30.91
02Teller App FAQ0.84
03Error Code Reference0.79
04End-of-Day Procedure0.74
05Branch Device Manual0.69
Open the Teller menu, select Reconcile Session, and clear the pending E-104 lock before retrying the transaction.
Operations Guide v3p. 12
15–30 min→<5 s
with page citationsmeasured

Business Impact

Figures labeled 'projected' or 'estimated' reflect business model targets rather than realized metrics.
measured
15–30 min→<5 s

Time to retrieve operational solution

Includes document title and exact page citations

measured
62%→100%

Out-of-scope refusal rate (hallucination defense)

Strict rejection of ungrounded queries

measured
0.78→1.00

Top-1 retrieval precision after reranking (Hit@1)

Empirical benchmark across custom eval suite

measured
Baseline→-13%

End-to-end query latency

Reduced LLM calls per query from 4 to 3

projected
>1,000 / mo→50–60% reduction

Routine service desk ticket volume

Targeted operational deflection

measured
220 / 340 pages→340 / 340 pages

Large PDF ingestion fidelity (340 pages)

Whole document now processed in full

The Problem

Operational staff and branch officers kept asking how to operate banking applications, what an error message meant, and how to run end-of-day procedures. Finding the answer meant searching operational guidelines, FAQs, and manuals, many of them full of screenshots.

Turnaround time for simple procedure questions routinely took 15 to 30 minutes, congesting central IT helpdesk queues with over a thousand repetitive tickets each month.

The Solution

We modernized and re-architected the ingestion, storage, and retrieval pipeline into an enterprise RAG standard:

  1. Layout-aware parsing: layout-aware chunking, with different parsing strategies for screenshot-heavy guides and text-dominant documents.
  2. Vector Search & Reranking: vector search over the indexed chunks, followed by a reranking step that orders candidates before answering.
  3. Knowledge Staging & Versioning: knowledge updates are verified on staging and tested against an evaluation set before promotion.
  4. Score-based refusal: when retrieved context falls below a relevance threshold, the assistant declines instead of guessing.

Business Impact

Operational staff previously spent 15 to 30 minutes manually searching dense manuals for routine procedures, generating over 1,000 monthly tickets at the IT service desk. Delivering grounded answers with document and page citations in under 5 seconds enables 24/7 self-service across branches and operations, projected to deflect 50–60% of routine inquiries and return an estimated ~633 working hours per month across the staff looking for answers (projected).

Engineering Principle

Treat Knowledge Like Code

Knowledge bases should be managed with the same rigor as production software. Knowledge updates are tested on staging against an evaluation set before promotion, with one-click rollback capabilities.

Behind the Decision

Behind the Decision

We evaluated a popular RAG technique, Hypothetical Document Embeddings (HyDE), and turned it off because multi-run evaluation showed it lowered the score by 8.5 points on the policy corpus. A low-recall case traced back to the default HNSW ef_search of 40, which we raised to 200 per query. On our evaluation set, out-of-scope refusal went from 62% to 100%, latency dropped 13%, and LLM calls per answer fell from 4 to 3 (measured).

Interested in shipping similar systems?

I partner with teams to design, evaluate, and ship production-grade AI systems with verified operational impact.

Let's talk on LinkedIn