MediBot Agentic RAG
Deterministic tool-routed medical RAG system with local retrieval, session memory, safety fallbacks, and benchmark telemetry.
MediBot is an educational symptom-checking prototype built around LangGraph ReAct orchestration, registered medical tools, TF-IDF + FAISS lexical retrieval, provider-neutral model adapters, conversation memory, safety fallbacks, Gradio, Docker, and pytest coverage.
Evidence
Case Study Highlights
60
safety, retrieval, and routing evaluation cases
86.67%
tool-selection accuracy
97.67%
lexical recall@k
100%
out-of-scope detection
0%
harmful response rate in eval
Context
Problem and ownership
The goal was to build a symptom-checking assistant that makes each medical step inspectable: which symptoms were extracted, which local records were retrieved, which tools ran, why safety declined a request, and what the evaluation suite verified. The project stays educational and corpus-bound instead of presenting itself as a clinical system.
My role
Designed the retrieval pipeline, deterministic tool-routing adapter, LangGraph orchestration layer, safety policy, evaluation harness, CI workflow, and local review interface.
Constraints
What shaped the solution
- The application is educational only and must never present itself as a clinician or medical device.
- Default operation needed to be deterministic, offline, and repeatable without provider API keys.
- Retrieval, routing, and safety behavior needed measurable outcomes rather than subjective screenshots.
- Non-medical, ambiguous, unsupported, and red-team prompts needed to be rejected or bounded before unsafe generation.
- Follow-up turns needed session memory so new symptoms could be evaluated with prior context.
Architecture
System design and request flow
MediBot is built as a staged pipeline: structured CSV datasets are normalized into disease profiles, retrieval documents are indexed with TF-IDF vectors stored in FAISS, four medical functions are exposed as LangChain tools, and a LangGraph ReAct orchestrator routes each turn through the required tool sequence.
Request and Data Flow
- 1CSV symptom and disease datasets
- 2Data cleaning, EDA outputs, and disease profile generation
- 3Query normalization, symptom aliases, and TF-IDF vectorization
- 4L2-normalized vectors persisted in a FAISS IndexFlatIP store
- 5Specialized diagnostic, severity, description, and precaution tools
- 6Provider-neutral model adapter with deterministic mock routing by default
- 7LangGraph ReAct orchestrator with session memory and pre-routing safety checks
- 8Gradio chat UI with thought/action trace
Core Components
- Data pipeline normalizes wide disease/symptom tables, severity weights, descriptions, precautions, and analytics artifacts.
- Vector service builds disease profiles, symptom profiles, and symptom evidence records, then persists FAISS, vectorizer, metadata, and manifest files.
- Agent tools isolate diagnosis lookup, urgency scoring, disease explanation, and precaution retrieval behind testable Python functions and LangChain tool wrappers.
- Orchestrator applies pre-routing refusals, invokes the LangGraph ReAct loop, formats trace events, records telemetry, and appends the required clinical disclaimer.
- Adapter layer supports deterministic mock mode plus lazy OpenAI, Anthropic, and OpenAI-compatible local providers without forcing cloud SDK initialization.
Technical decisions
Key choices and why they mattered
Used LangGraph ReAct with registered tools instead of a fixed chain
The same query can require diagnosis, severity triage, explanations, precautions, or a dependency-ordered combination. ReAct routing keeps those paths explicit in the trace instead of hiding them inside one prompt.
Used TF-IDF + FAISS lexical retrieval as the default backend
The corpus is bounded and symptom-heavy, so lexical matching with query expansion gives deterministic local behavior, inspectable rankings, and zero API dependency for review.
Separated plain Python behavior from LangChain tool wrappers
The same behavior can be tested directly without relying on a running chat loop, making evaluation and debugging simpler.
Added a deterministic mock model adapter
The default adapter produces repeatable tool calls, latency telemetry, and evaluation results without loading provider SDKs or requiring API keys.
Trade-offs
What the design optimizes for and gives up
- Lexical retrieval improves determinism and local repeatability, but unfamiliar wording can still under-retrieve without aliases or query expansion.
- The disease corpus is bounded by the local dataset, so unsupported medical knowledge must be declined instead of inferred.
- Strict refusal rules reduce hallucination risk, but they also reject vague prompts that might be answerable in a broader clinical assistant.
- Provider-backed adapters can be swapped in, but each provider/model configuration needs its own evaluation run because tool sequences and latency can change.
Production readiness
Reliability, security, and delivery evidence
- GitHub Actions runs pytest with coverage on pushes and pull requests to main and dev using LLM_PROVIDER=mock.
- Automated tests cover adapters, cache behavior, isolated agents, memory/orchestrator behavior, evaluation metrics, and safety fallbacks.
- Docker packaging uses a repeatable Gradio run path with environment-variable configuration.
- Every final answer includes the clinical disclaimer, and out-of-scope requests are declined before diagnostic tools run.
- Evaluation scripts produce routing, retrieval, faithfulness, safety, latency, token, and cost telemetry across a fixed 60-case suite.
Results
What changed because of the work
- Verified 86.67% exact tool-selection accuracy across the deterministic evaluation suite.
- Reached 97.67% lexical recall@5 and 70.7% lexical precision@5.
- Measured 100% answer faithfulness, 0% unsupported-claim rate, and 100% answer relevance in the benchmark report.
- Maintained 100% out-of-scope detection, 100% abstention accuracy, 0% safety false-positive rate, 0% safety false-negative rate, and 0% harmful response rate in the tested cases.
- Recorded local latency at 25.251 ms p50 and 976.412 ms p95 in mock mode with $0.00 estimated cost per evaluated case.
Next improvements
Where I would take it next
- Add dense retrieval as an optional second-stage reranker and compare failure modes against lexical retrieval.
- Expand the medical dataset and add provenance display for every retrieved fact.
- Persist trace and telemetry events to an observability sink instead of only showing the latest run in the Gradio UI.