RAG Observability and Evaluations

RAG Observability and Evaluations

Run retrieval-augmented generation like a measurable production system, not a black box.

When to Use This Skill

• Deploying a RAG system to production and need quality monitoring
• Setting up automated evaluation pipelines for retrieval and generation
• Debugging hallucination or relevance regressions
• Building dashboards for RAG-specific golden signals
• Establishing quality gates for RAG pipeline changes

Prerequisites

• RAG pipeline with instrumented retrieval and generation stages
• Python 3.10+ with evaluation libraries (ragas, langchain, openai)
• Prometheus endpoint for custom metrics export
• Benchmark dataset with gold-standard question/answer/source triples
• OpenTelemetry SDK integrated into the RAG service

What to Measure

Retrieval Quality

• Recall@k and MRR for top-k chunks
• Citation coverage and source freshness
• Embedding drift and index staleness

Generation Quality

• Groundedness score (answer supported by retrieved context)
• Hallucination rate by route/use case
• Instruction adherence and format validity

Reliability and Cost

• p50/p95 latency split by retrieval vs generation
• Token usage per stage
• Cache hit rate and cost per successful answer

RAGAS Evaluation Script