MediBot Agentic RAG
Agentic RAG medical assistant with tool routing, memory, retrieval, safety controls, and adversarial evaluation.
MediBot is an educational medical symptom-checking prototype that combines LangGraph ReAct orchestration, specialized tools, lexical FAISS retrieval, conversational memory, safety fallbacks, Gradio, Docker, and automated tests.
Evidence
Case Study Highlights
60
adversarial test cases
97.67%
lexical recall@k
100%
out-of-scope detection
0%
harmful response rate in eval
Context
Problem and ownership
The goal was to build a medical assistant prototype that could answer symptom-related questions using local, inspectable data instead of relying on opaque retrieval or unbounded model behavior. The project needed to show agentic AI engineering while staying honest about medical risk.
My role
Designed and implemented the agent orchestration, retrieval layer, safety behavior, evaluation workflow, and local deployment path.
Constraints
What shaped the solution
- The application is educational only and must never present itself as a clinician or medical device.
- Retrieval and evaluation needed to run locally for deterministic reviewer verification.
- The system needed to reject non-medical or unsupported prompts before diagnostic tooling could run.
- Follow-up turns needed memory so users could add symptoms without starting a new conversation.
Architecture
System design and request flow
MediBot is built as a staged pipeline: dataset ingestion creates normalized disease profiles, lexical retrieval indexes them, specialized tools expose medical functions, and a LangGraph ReAct orchestrator decides which tools to call.
Request and Data Flow
- 1CSV symptom and disease datasets
- 2Data cleaning and disease profile generation
- 3TF-IDF vectorization persisted in FAISS
- 4Specialized diagnostic, severity, description, and precaution tools
- 5LangGraph ReAct orchestrator with memory and safety prompt
- 6Gradio chat UI with thought/action trace
Core Components
- Data pipeline normalizes symptom names, severity weights, descriptions, and precautions.
- Vector service builds lexical documents and persists FAISS artifacts for repeatable retrieval.
- Agent tools isolate diagnosis support, urgency estimation, condition explanation, and precaution guidance.
- Orchestrator enforces safety checks, memory, tool routing, and final clinical disclaimers.
Technical decisions
Key choices and why they mattered
Used LangGraph ReAct instead of a fixed chain
ReAct routing lets the assistant choose different tool sequences for diagnosis, triage, explanations, and precautions instead of forcing every prompt through the same path.
Used lexical FAISS retrieval instead of embedding APIs
Local lexical retrieval is cheaper, deterministic, easy to inspect, and suitable for a bounded symptom vocabulary during portfolio review.
Kept plain Python tool functions beside LangChain tools
The same behavior can be tested directly without relying on a running chat loop, making evaluation and debugging simpler.
Trade-offs
What the design optimizes for and gives up
- Lexical retrieval improves determinism and local repeatability, but it has weaker semantic recall than dense embeddings for unfamiliar phrasing.
- The disease corpus is bounded by the local dataset, so unsupported medical knowledge must be declined instead of inferred.
- The prototype prioritizes safety and explainability over broad medical coverage.
Production readiness
Reliability, security, and delivery evidence
- Automated tests cover isolated agents, memory behavior, ReAct routing, lexical retrieval, and safety fallbacks.
- Docker packaging gives reviewers a repeatable local run path.
- Every response includes a clinical disclaimer and out-of-scope requests are declined before tool execution.
- Evaluation scripts report retrieval and safety metrics against a 60-case adversarial suite.
Results
What changed because of the work
- Reached 97.67% lexical recall@k in the documented evaluation suite.
- Maintained 100% out-of-scope detection in the tested safety cases.
- Kept harmful response rate at 0% in the adversarial evaluation.
- Demonstrated multi-turn memory and multi-tool routing through reviewer prompts.
Next improvements
Where I would take it next
- Add dense retrieval as an optional second-stage reranker and compare failure modes against lexical retrieval.
- Expand the medical dataset and add provenance display for every retrieved fact.
- Add structured observability for tool calls, fallback reasons, and retrieval misses.