LLMGuard Evaluation Framework
Repeatable AI safety evaluation framework for prompt injection, toxicity, topic, and PII filtering.
LLMGuard Eval tests prompt-injection, toxicity, topic, and PII scanners across integrated LLM workflows with scripts, notebook analysis, metrics, confusion matrices, and false-negative review.
Evidence
Case Study Highlights
3
model backends
5
classification metrics
4
scanner categories
1
reproducible notebook
Context
Problem and ownership
LLM applications need safety checks that can be evaluated, not just asserted. This project tests whether LLMGuard can identify prompt injection, toxicity, topic drift, and PII exposure risks before unsafe inputs or outputs move through an LLM workflow.
My role
Built the evaluation workflow, integrated model experiments, measured classification performance, and documented failure patterns.
Constraints
What shaped the solution
- The same defense workflow needed to run across multiple model and scanner configurations.
- Results needed standard classification metrics rather than anecdotal examples only.
- The workflow needed to preserve enough notebook context for reviewers to inspect methodology.
- False negatives needed explicit analysis because missed attacks are the most important safety failure mode.
Architecture
System design and request flow
Prompts and responses pass through LLMGuard scanners that assign risk, block or sanitize unsafe content, and produce predictions that can be compared against labeled evaluation data.
Request and Data Flow
- 1Labeled safety datasets
- 2Prompt injection, toxicity, topic, and PII scanners
- 3Risk scoring
- 4Accept, reject, or sanitize decision
- 5Model response path
- 6Metrics and confusion matrix reporting
Core Components
- Separate scripts evaluate TinyLLaMA, Phi-2, and DistilGPT-2 workflows.
- Notebook walkthrough shows model loading, scanner usage, manual tests, and dataset evaluation.
- Metrics include accuracy, precision, recall, F1, classification report, and confusion matrix.
- Safety coverage includes prompt injection, toxicity, topic control, PII exposure, instruction override, roleplay injection, malicious payloads, and context confusion.
Technical decisions
Key choices and why they mattered
Used a repeatable scanner-before-inference pipeline
The safety decision is isolated from the model output path, making it easier to evaluate and compare across backends.
Measured classification metrics instead of only demo prompts
Metrics reveal false positives, false negatives, and class-level behavior that a few hand-picked examples would hide.
Kept notebook and script workflows
Scripts support reproducibility while the notebook keeps the methodology and visual analysis easy to review.
Trade-offs
What the design optimizes for and gives up
- Scanner-based guardrails are easier to integrate than retraining, but they can still miss novel injection patterns.
- Open lightweight models are practical for local experiments, but they do not represent every production LLM behavior.
- Strong blocking improves safety but may reject benign prompts when the scanner is too conservative.
Production readiness
Reliability, security, and delivery evidence
- Experiment scripts separate model-specific runs and make repeated evaluation practical.
- The notebook documents methodology, metrics, visualizations, and failure analysis.
- The workflow reports confusion matrices and classification reports for reviewer inspection.
- Scanner categories and attack patterns are documented to make coverage boundaries clear.
Results
What changed because of the work
- Created a repeatable evaluation story across TinyLLaMA, Phi-2, and DistilGPT-2.
- Documented prompt-injection, toxicity, topic, and PII scanner behavior with standard ML classification metrics.
- Added false-negative analysis so limitations are visible instead of hidden.
Next improvements
Where I would take it next
- Add a larger and more diverse adversarial prompt dataset.
- Track results over time with versioned evaluation reports.
- Add latency and cost measurements for scanner overhead.