A RAG pipeline that answers questions in the demo is not the same thing as a RAG pipeline that answers them correctly.
The gap between the two only shows up once you start measuring precision, recall, and faithfulness instead of eyeballing a handful of outputs.
This post covers two ways to run that measurement: a fully local setup with a vector database and an open model as judge, and Amazon Bedrock's native Knowledge Bases evaluation.
How the evaluation knows what is correct
This is the part that trips people up, and it splits the metrics into two families. Faithfulness and answer relevancy do not need a labeled correct answer at all. For faithfulness, the judge LLM breaks the generated answer into individual factual claims, then checks each one against the retrieved context: a claim not supported by anything retrieved counts against the score. For answer relevancy, the judge generates a handful of questions that the answer would plausibly be responding to, then compares those against the original question using embedding similarity. Both checks are self-contained: question, context, and answer, nothing else.
Context recall, and usually context precision, work differently and do need a labeled example. You have to supply a ground truth, meaning a correct reference answer written by you or a subject matter expert ahead of time. The judge decomposes that reference answer into statements and checks whether each one is backed by something in the retrieved context. If the reference says three facts and only two show up in what got retrieved, context recall lands at roughly 0.67, regardless of what the model went on to generate. This is why the practical bottleneck in either approach below is not the tooling, it is building a small set of representative questions with a correct answer already written for each one. Fifteen to thirty well chosen pairs, covering both easy and edge case queries, will tell you more than a hundred pairs nobody reviewed. Everything else in this post assumes you already have that set.
What you are actually measuring
RAG evaluation splits into two layers, and conflating them is the most common mistake. The retrieval layer asks whether the right chunks came back from the vector search: context precision (how many of the retrieved chunks are relevant) and context recall (how many of the relevant chunks got retrieved). The generation layer asks whether the model did something sensible with those chunks: faithfulness (is the answer grounded in the retrieved text, or invented) and answer relevancy (does the answer actually address the question). You can have perfect retrieval and a hallucinating generator, or a great generator fed garbage context. Measuring only the end-to-end answer hides which half is broken.
Both approaches below compute these same four metrics. What differs is where the judging happens and what it costs you to get there.
 |
| Diagram showing RAG evaluation and metrics |
Approach 1: Local vector database with an open model as judge
This setup runs a vector store (Chroma, in the example below) and a local LLM through Ollama, then hands both to RAGAS to compute the scores. Nothing leaves your machine, and there is no per-token bill. If your embeddings already live inside a relational database rather than a standalone store, generating them directly from SQL against Bedrock is worth a look before you stand up a separate pipeline just for this evaluation.
Pros: zero API cost per evaluation run, full data privacy for sensitive corpora, no dependency on AWS region availability, easy to iterate quickly on chunking or embedding changes since a run costs only compute time.
Cons: judge quality depends on the local model, and smaller open models are noticeably less reliable graders than Claude or GPT-4 class models. RAGAS issues with Ollama timeouts are a known friction point on CPU-only machines. You also own the entire pipeline: embeddings, chunking, retriever tuning, and judge prompt behavior, with no managed comparison view across runs.
A minimal setup looks like this. Install the pieces first:
# local stack: chroma, sentence embeddings, ollama-backed judge
pip install ragas chromadb sentence-transformers openai
ollama pull llama3.1
ollama pull nomic-embed-text
Build the retriever over your own documents, run a handful of test questions through it, and collect question, retrieved contexts, generated answer, and (if you have them) reference answers into a dataset. Then point RAGAS at your local Ollama server through its OpenAI-compatible endpoint:
-- pseudocode-style Python, adjust names to your pipeline from openai import OpenAI from ragas.llms import llm_factory from ragas import evaluate from ragas.metrics import ( context_precision, context_recall, faithfulness, answer_relevancy, ) from datasets import Dataset client = OpenAI(api_key="ollama", base_url="http://localhost:11434/v1") judge = llm_factory("llama3.1", provider="openai", client=client) eval_set = Dataset.from_dict({ "question": questions, "contexts": retrieved_contexts, "answer": generated_answers, "ground_truth": reference_answers, }) result = evaluate( eval_set, metrics=[context_precision, context_recall, faithfulness, answer_relevancy], llm=judge, ) print(result)
The output is a dictionary of scores between 0 and 1 for each metric, and calling result.to_pandas() gives you a per-question breakdown so you can see exactly which retrievals or answers dragged the average down. Swap Chroma for FAISS or Qdrant, or the embedding model for a different sentence-transformers checkpoint, rerun, and compare the numbers directly. If you are deciding between a dedicated vector store like this and keeping vectors inside a relational engine, the trade-offs across SQL Server and PostgreSQL's native vector support are worth reading before you commit to either path.
Approach 2: Amazon Bedrock Knowledge Bases evaluation
Bedrock has a built-in evaluation job type for Knowledge Bases, generally available since March 2025, that uses a foundation model as judge and computes the metrics for you, with no RAGAS or custom scoring code required. It supports retrieval-only evaluation, full retrieve-and-generate evaluation, and evaluating a RAG system hosted anywhere by supplying your own inference responses in the input dataset. Citation precision and citation coverage were added later to check whether generated answers are actually grounded in the sources they cite.
Pros: no evaluation infrastructure to build or maintain, console comparison view across multiple evaluation runs (useful when testing different chunking strategies or embedding models side by side), stronger judge models available (Claude models as evaluator), and you can evaluate a knowledge base you already run in production without extra tooling.
Cons: you pay standard Bedrock on-demand pricing for both the evaluator model and the generator model on every run, it is currently limited to a subset of AWS regions, the service is optimized for English content, and your evaluation data has to live in S3 with a specific JSONL structure, which adds setup overhead compared to a Python dict.
The evaluation dataset is a JSONL file where each line carries a question and a reference answer:
{"conversationTurns":[{
"referenceResponses":[{"content":
[{"text":"A trigger invokes a Lambda function."}]}],
"prompt":{"content":
[{"text":"What is a Lambda trigger?"}]}
}]}
From the console, under Evaluations, Knowledge Bases, you create a job, pick the evaluator model, choose whether to evaluate retrieval only or retrieval plus generation, select the metrics, and point it at the S3 dataset. The same job can be created through boto3 for a repeatable pipeline:
import boto3 bedrock = boto3.client("bedrock", region_name="us-east-1") bedrock.create_evaluation_job( jobName="kb-rag-eval-run-1", roleArn="arn:aws:iam::ACCOUNT_ID:role/BedrockEvalRole", applicationType="RagEvaluation", evaluationConfig={ "automated": { "datasetMetricConfigs": [{ "taskType": "General", "dataset": { "name": "kb_eval_set", "datasetLocation": { "s3Uri": "s3://my-bucket/eval-set.jsonl" } }, "metricNames": [ "Builtin.ContextCoverage", "Builtin.ContextRelevance", "Builtin.Faithfulness", "Builtin.Correctness", ] }], "evaluatorModelConfig": { "bedrockEvaluatorModels": [{ "modelIdentifier": "anthropic.claude-3-5-sonnet-20241022-v2:0" }] } } }, inferenceConfig={ "ragConfigs": [{ "knowledgeBaseConfig": { "retrieveAndGenerateConfig": { "knowledgeBaseId": "KB_ID", "modelArn": "anthropic.claude-3-haiku-20240307-v1:0" } } }] }, outputDataConfig={"s3Uri": "s3://my-bucket/eval-output/"} )
The exact metric names and payload shape depend on whether you are running a retrieval-only or a retrieve-and-generate job, so check the current Bedrock knowledge base evaluation documentation before wiring this into a pipeline. Once the job finishes, the console gives you a metric summary, a per-metric breakdown with the judge's reasoning for each score, and a side by side comparison view if you run the same dataset against two different knowledge base configurations, which is the part a local RAGAS script does not give you out of the box.
Which one to reach for
Local with RAGAS and Ollama fits fast iteration during development, sensitive or regulated corpora that cannot leave your environment, and situations where the evaluation itself has to run at effectively no marginal cost. Bedrock's native evaluation fits teams already running Knowledge Bases in production, cases where you want a stronger judge model without hosting one, and any workflow where comparing several configurations side by side matters more than shaving cost. Nothing stops you from running both: prototype and tune locally, then confirm the final numbers with Bedrock's evaluator before shipping.
