How to Automate RAG Regression Testing and Hallucination Checks in GitHub Actions Using Ragas and Ollama

Your RAG app passed every test on Friday. On Monday, a customer asks the support bot about refunds and it confidently invents a 90-day policy that does not exist. Nothing crashed. No exception was logged. Latency looked great. The only thing that changed was a "harmless" pull request that tweaked the chunk size and rewrote a sentence in the system prompt.

That is the blind spot of shipping AI features: silent quality degradation. Traditional tests check that code runs. They cannot tell you whether your retrieval still finds the right paragraph or whether your model is still sticking to the facts it was given.

In this tutorial, we will build a production-style safety net. By the end, you will have a GitHub Actions workflow that:

• Spins up a containerized Ollama instance inside the CI runner
• Runs your RAG pipeline against a golden dataset on every pull request
• Scores the results with the Ragas framework (faithfulness, answer relevancy, context recall)
• Compares them against absolute thresholds and a committed baseline
• Blocks the merge when hallucinations or retrieval drift sneak in

No paid API keys. No data leaving your infrastructure. Just a repeatable quality gate that treats your RAG system like the critical software it has become.

Why RAG Systems Need Regression Testing

Retrieval-Augmented Generation has more moving parts than a plain LLM call, and every one of them can regress independently. A typical pipeline has:

1. Ingestion: parsing, cleaning, and chunking documents
2. Embedding: turning chunks and queries into vectors
3. Retrieval: top-k search, filtering, maybe re-ranking
4. Prompting: the instructions and context template
5. Generation: the model that writes the final answer

Change any one of these and the whole system's behavior shifts. Here are the changes that most often cause trouble in real teams:

• Chunking tweaks. Going from 900 to 400 characters can split a policy sentence from its exception clause.
• Embedding model swaps. A "better" model on a leaderboard may perform worse on your domain vocabulary.
• Prompt edits. Removing one line like "answer only from the context" is enough to let the model improvise.
• Corpus updates. New documents can crowd out the right chunks in the top-k results.
• Dependency upgrades. A minor version bump in a vector library can change scoring or tie-breaking.

The nasty part is that none of these produce errors. The system keeps answering, and the answers just get subtly worse. Without an automated gate, you find out from users.

Regression testing for RAG is the same idea as regression testing for code: freeze a set of expected behaviors, run them on every change, and fail the build when behavior gets worse.

The Architecture We Are Building

Here is the full flow, from pull request to merge decision:

Pull Request
     |
     v
GitHub Actions runner (ubuntu-latest)
     |
     +--> Docker: ollama/ollama  (generation, judge, embedding models)
     |
     +--> Python: RAG pipeline answers each golden question
     |
     +--> Deterministic retrieval check (no LLM needed)
     |
     +--> Ragas: faithfulness, answer relevancy, context recall
     |
     +--> Gate: absolute floors + baseline tolerance + NaN budget
     |
     v
PR comment + job summary + pass/fail status check

The key design decision is to use Ollama for everything: the model that generates answers, the model that judges them, and the embedding model. That keeps the pipeline free, private, and deterministic enough to trust.

Understanding the Metrics

Before writing code, it helps to know exactly what each number means, because you will be defending these thresholds in code review.

Faithfulness (your hallucination detector)

Faithfulness measures whether the claims in the generated answer are supported by the retrieved context. Ragas breaks the answer into individual statements, then asks the judge model whether each statement can be inferred from the context. The score is the fraction of supported statements.

If the answer says "refunds take 90 days" and no retrieved chunk says that, faithfulness drops. This is the metric that catches the Monday-morning disaster from the intro.

Answer Relevancy (did it answer the question?)

Answer relevancy checks whether the response is actually on topic. A response can be perfectly faithful and still useless, such as a correct paragraph about shipping when the user asked about refunds. Ragas generates likely questions from the answer and measures how similar they are to the original question using embeddings.

Context Recall (your retrieval drift detector)

Context recall compares the reference answer from your golden dataset with the retrieved chunks and asks: was the information needed to produce this answer actually retrieved? When someone changes chunking, top-k, or the embedding model and the right passage stops showing up, this is the metric that drops.

A cheap bonus: deterministic retrieval hit rate

LLM-judged metrics are powerful but noisy and slow. For each golden question we can also store the source document that must appear in the retrieved set. Checking that is plain Python: no model, no flakiness. It makes an excellent first-line gate.

Prerequisites and Project Layout

You will need Python 3.11 or newer, Docker for local testing (GitHub runners already have it), and a repository on GitHub. This is the layout we will use:

rag-app/
├── app/
│   ├── __init__.py
│   └── rag.py                  # the RAG pipeline under test
├── docs/
│   ├── refunds.md              # knowledge base documents
│   └── shipping.md
├── evals/
│   ├── golden.jsonl            # golden test dataset
│   ├── thresholds.json         # merge gate rules
│   ├── baseline.json           # last accepted scores
│   └── run_eval.py             # evaluation runner
├── requirements.txt
├── requirements-eval.txt
└── .github/workflows/rag-eval.yml

Install the evaluation dependencies. Pin exact versions in your lockfile, because the Ragas API has evolved quickly and you want CI to fail because of your change, not because a library moved underneath you:

# requirements-eval.txt
ragas==0.3.*
langchain-ollama
ollama
numpy
pandas

Heads up: The imports in this article target the Ragas 0.3 series. If you pin a different major or minor version, double-check the class names against the Ragas documentation before copying anything.

Step 1: A Minimal RAG Pipeline to Test

To keep the tutorial focused, here is a deliberately small pipeline built on the Ollama Python client and NumPy. The evaluation approach works the same if your production system uses LangChain, LlamaIndex, or a hosted vector database. All that matters is that you can call one function and get back an answer plus the contexts used to produce it.

# app/rag.py
from __future__ import annotations

import os
import pathlib
from dataclasses import dataclass

import numpy as np
import ollama

OLLAMA_HOST = os.getenv("OLLAMA_HOST", "http://127.0.0.1:11434")
GEN_MODEL = os.getenv("GEN_MODEL", "llama3.2:3b")
EMBED_MODEL = os.getenv("EMBED_MODEL", "nomic-embed-text")
TOP_K = int(os.getenv("RAG_TOP_K", "4"))
CHUNK_CHARS = int(os.getenv("RAG_CHUNK_CHARS", "900"))

SYSTEM_PROMPT = (
    "You are a support assistant. Answer the question using ONLY the "
    "context provided. If the context does not contain the answer, reply "
    "exactly: I don't know based on the available documentation. "
    "Keep answers short and factual."
)


@dataclass
class Chunk:
    source: str
    text: str


@dataclass
class RagResult:
    answer: str
    contexts: list[str]
    sources: list[str]


def load_chunks(docs_dir: str) -> list[Chunk]:
    """Split each markdown file on blank lines, then merge to ~CHUNK_CHARS."""
    chunks: list[Chunk] = []
    for path in sorted(pathlib.Path(docs_dir).glob("*.md")):
        buffer = ""
        for paragraph in path.read_text(encoding="utf-8").split("\n\n"):
            paragraph = paragraph.strip()
            if not paragraph:
                continue
            if buffer and len(buffer) + len(paragraph) > CHUNK_CHARS:
                chunks.append(Chunk(path.name, buffer))
                buffer = ""
            buffer = f"{buffer}\n\n{paragraph}".strip()
        if buffer:
            chunks.append(Chunk(path.name, buffer))
    return chunks


class RagPipeline:
    def __init__(self, docs_dir: str = "docs") -> None:
        self.client = ollama.Client(host=OLLAMA_HOST)
        self.chunks = load_chunks(docs_dir)
        self.matrix = self._embed([c.text for c in self.chunks])

    def _embed(self, texts: list[str]) -> np.ndarray:
        response = self.client.embed(model=EMBED_MODEL, input=texts)
        vectors = np.array(response["embeddings"], dtype=np.float32)
        return vectors / np.linalg.norm(vectors, axis=1, keepdims=True)

    def retrieve(self, question: str) -> list[Chunk]:
        query = self._embed([question])[0]
        scores = self.matrix @ query
        top = np.argsort(-scores)[:TOP_K]
        return [self.chunks[i] for i in top]

    def answer(self, question: str) -> RagResult:
        retrieved = self.retrieve(question)
        context = "\n\n---\n\n".join(c.text for c in retrieved)
        response = self.client.chat(
            model=GEN_MODEL,
            messages=[
                {"role": "system", "content": SYSTEM_PROMPT},
                {
                    "role": "user",
                    "content": f"Context:\n{context}\n\nQuestion: {question}",
                },
            ],
            options={"temperature": 0, "seed": 42},
        )
        return RagResult(
            answer=response["message"]["content"].strip(),
            contexts=[c.text for c in retrieved],
            sources=sorted({c.source for c in retrieved}),
        )

Notice two small choices that matter a lot for CI. temperature is 0 and seed is fixed, so repeated runs on the same commit produce near-identical output. And every tunable (top-k, chunk size, models) reads from environment variables, so you can run experiments without editing code.

Step 2: Build a Golden Dataset

The golden dataset is the heart of your regression suite. It is a versioned file of questions, the reference answers a human has verified, and the source documents that should be retrieved. Store it as JSONL so diffs are readable in pull requests:

{"question": "How many days do I have to request a refund?", "reference": "You can request a refund within 30 days of purchase.", "expected_sources": ["refunds.md"]}
{"question": "Are digital gift cards refundable?", "reference": "Digital gift cards are non-refundable once the code has been redeemed.", "expected_sources": ["refunds.md"]}
{"question": "How long does standard shipping take?", "reference": "Standard shipping takes 3 to 5 business days within the continental US.", "expected_sources": ["shipping.md"]}
{"question": "Do you offer phone support on weekends?", "reference": "I don't know based on the available documentation.", "expected_sources": []}

That last row is easy to overlook and extremely valuable. It is an out-of-scope question whose correct behavior is refusing to answer. A pipeline that starts inventing weekend support hours is hallucinating, and this test catches it.

What makes a good golden dataset

• Cover real intents. Pull questions from support tickets, search logs, or user interviews rather than inventing them at your desk.
• Include the awkward cases. Add multi-step questions, questions with exceptions, ambiguous wording, and questions with no answer in the corpus.
• Keep references short and verifiable. A reference that is one or two sentences is easier for a judge model to compare against.
• Version it like code. Every change to the dataset goes through review, so nobody quietly lowers the bar.
• Grow it from incidents. Each production hallucination becomes a new permanent test row.

Aim for 30 to 50 rows to start. On CPU-only runners, that keeps a full evaluation within a reasonable time budget.

Step 3: Score the Pipeline with Ragas and Ollama

Ragas normally defaults to OpenAI as its judge. We will point it at Ollama by wrapping LangChain's Ollama classes. The judge model and the embedding model both run locally in the container.

# evals/run_eval.py
from __future__ import annotations

import json
import os
import pathlib
import sys

from langchain_ollama import ChatOllama, OllamaEmbeddings
from ragas import EvaluationDataset, evaluate
from ragas.embeddings import LangchainEmbeddingsWrapper
from ragas.llms import LangchainLLMWrapper
from ragas.metrics import Faithfulness, LLMContextRecall, ResponseRelevancy
from ragas.run_config import RunConfig

from app.rag import RagPipeline

OLLAMA_HOST = os.getenv("OLLAMA_HOST", "http://127.0.0.1:11434")
JUDGE_MODEL = os.getenv("JUDGE_MODEL", "qwen2.5:7b-instruct")
EMBED_MODEL = os.getenv("EMBED_MODEL", "nomic-embed-text")

GOLDEN_PATH = pathlib.Path("evals/golden.jsonl")
THRESHOLDS_PATH = pathlib.Path("evals/thresholds.json")
BASELINE_PATH = pathlib.Path("evals/baseline.json")
REPORT_DIR = pathlib.Path("eval-report")

RAGAS_METRICS = ["faithfulness", "answer_relevancy", "context_recall"]


def load_jsonl(path: pathlib.Path) -> list[dict]:
    with path.open(encoding="utf-8") as handle:
        return [json.loads(line) for line in handle if line.strip()]


def run_pipeline(golden: list[dict]) -> tuple[list[dict], float]:
    """Answer every golden question and compute the deterministic hit rate."""
    rag = RagPipeline("docs")
    rows: list[dict] = []
    hits: list[bool] = []

    for item in golden:
        result = rag.answer(item["question"])
        expected = set(item.get("expected_sources", []))
        if expected:
            hits.append(expected.issubset(set(result.sources)))
        rows.append(
            {
                "user_input": item["question"],
                "retrieved_contexts": result.contexts,
                "response": result.answer,
                "reference": item["reference"],
            }
        )

    hit_rate = sum(hits) / len(hits) if hits else 1.0
    return rows, hit_rate


def score_with_ragas(rows: list[dict]):
    judge = LangchainLLMWrapper(
        ChatOllama(
            model=JUDGE_MODEL,
            base_url=OLLAMA_HOST,
            temperature=0,
            num_ctx=8192,
        )
    )
    embeddings = LangchainEmbeddingsWrapper(
        OllamaEmbeddings(model=EMBED_MODEL, base_url=OLLAMA_HOST)
    )

    result = evaluate(
        dataset=EvaluationDataset.from_list(rows),
        metrics=[Faithfulness(), ResponseRelevancy(), LLMContextRecall()],
        llm=judge,
        embeddings=embeddings,
        run_config=RunConfig(timeout=300, max_workers=2, max_retries=3),
        raise_exceptions=False,
        show_progress=True,
    )
    return result.to_pandas()


def aggregate(frame) -> tuple[dict, dict]:
    """Average each metric, ignoring NaN, and report how many were NaN."""
    scores, nan_rates = {}, {}
    for metric in RAGAS_METRICS:
        column = frame[metric]
        valid = column.dropna()
        scores[metric] = round(float(valid.mean()), 4) if len(valid) else 0.0
        nan_rates[metric] = round(1 - len(valid) / len(column), 4)
    return scores, nan_rates

A few things worth explaining here:

• raise_exceptions=False lets the evaluation continue if the judge fails on a single row. Instead of crashing, Ragas returns NaN for that score, and we track the NaN rate separately.
• max_workers=2 keeps a CPU-bound Ollama from being flooded with parallel requests. On small runners, more workers usually means slower, not faster.
• num_ctx=8192 gives the judge enough room for long contexts. Ollama's default context window can silently truncate prompts, which quietly corrupts scores.
• NaN handling matters. Smaller judge models sometimes return malformed JSON that Ragas cannot parse. If you average over NaN values without tracking them, a broken judge can make a bad pull request look fine.

Step 4: Define the Merge Gate

Now the decision logic. We use three complementary rules, because each catches something the others miss:

1. Absolute floors ensure quality never drops below a minimum acceptable level.
2. Baseline tolerance catches gradual erosion. A score can stay above the floor and still be sliding downhill.
3. NaN budget makes sure the judge itself is healthy.

{
  "absolute_floor": {
    "faithfulness": 0.80,
    "answer_relevancy": 0.75,
    "context_recall": 0.70,
    "retrieval_hit_rate": 0.85
  },
  "max_regression_vs_baseline": 0.05,
  "max_nan_rate": 0.10
}

And the code that enforces it, continuing run_eval.py:

def check_gates(scores: dict, nan_rates: dict, thresholds: dict, baseline: dict) -> list[str]:
    failures: list[str] = []

    for metric, floor in thresholds["absolute_floor"].items():
        if scores[metric] < floor:
            failures.append(
                f"{metric} is {scores[metric]:.2f}, below the floor of {floor:.2f}"
            )

    tolerance = thresholds["max_regression_vs_baseline"]
    for metric, previous in baseline.items():
        drop = previous - scores.get(metric, 0.0)
        if drop > tolerance:
            failures.append(
                f"{metric} regressed by {drop:.2f} vs baseline "
                f"({previous:.2f} to {scores[metric]:.2f})"
            )

    for metric, rate in nan_rates.items():
        if rate > thresholds["max_nan_rate"]:
            failures.append(
                f"{metric} had {rate:.0%} unscored rows (judge output could not be parsed)"
            )

    return failures


def render_report(scores, nan_rates, baseline, failures) -> str:
    lines = [
        "## RAG Evaluation Report",
        "",
        "| Metric | Score | Baseline | Delta |",
        "| --- | --- | --- | --- |",
    ]
    for metric, value in scores.items():
        base = baseline.get(metric)
        delta = f"{value - base:+.2f}" if base is not None else "n/a"
        base_text = f"{base:.2f}" if base is not None else "n/a"
        lines.append(f"| {metric} | {value:.2f} | {base_text} | {delta} |")

    lines.append("")
    if failures:
        lines.append("**Result: FAILED**")
        lines.extend(f"- {failure}" for failure in failures)
    else:
        lines.append("**Result: PASSED**")
    return "\n".join(lines)


def main() -> int:
    golden = load_jsonl(GOLDEN_PATH)
    thresholds = json.loads(THRESHOLDS_PATH.read_text())
    baseline = json.loads(BASELINE_PATH.read_text()) if BASELINE_PATH.exists() else {}

    rows, hit_rate = run_pipeline(golden)
    frame = score_with_ragas(rows)
    scores, nan_rates = aggregate(frame)
    scores["retrieval_hit_rate"] = round(hit_rate, 4)

    failures = check_gates(scores, nan_rates, thresholds, baseline)
    report = render_report(scores, nan_rates, baseline, failures)

    REPORT_DIR.mkdir(exist_ok=True)
    (REPORT_DIR / "report.md").write_text(report, encoding="utf-8")
    (REPORT_DIR / "scores.json").write_text(json.dumps(scores, indent=2))
    frame.to_csv(REPORT_DIR / "per_question.csv", index=False)

    summary_file = os.getenv("GITHUB_STEP_SUMMARY")
    if summary_file:
        with open(summary_file, "a", encoding="utf-8") as handle:
            handle.write(report + "\n")

    print(report)
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main())

The script writes three outputs: a human-readable Markdown report, a machine-readable scores.json (useful for updating the baseline), and a per-question CSV so reviewers can see which questions got worse. The non-zero exit code is what actually fails the CI job.

Step 5: The GitHub Actions Workflow

Here is the complete workflow. It starts Ollama in Docker, caches the model files so you do not re-download several gigabytes on every run, and executes the evaluation.

# .github/workflows/rag-eval.yml
name: RAG Evaluation

on:
  pull_request:
    paths:
      - "app/**"
      - "docs/**"
      - "evals/**"
      - "requirements*.txt"
      - ".github/workflows/rag-eval.yml"
  workflow_dispatch:

concurrency:
  group: rag-eval-${{ github.ref }}
  cancel-in-progress: true

permissions:
  contents: read
  pull-requests: write

env:
  OLLAMA_HOST: http://127.0.0.1:11434
  GEN_MODEL: llama3.2:3b
  JUDGE_MODEL: qwen2.5:7b-instruct
  EMBED_MODEL: nomic-embed-text

jobs:
  rag-eval:
    runs-on: ubuntu-latest
    timeout-minutes: 45

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
          cache-dependency-path: requirements-eval.txt

      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install -r requirements-eval.txt

      - name: Restore Ollama model cache
        id: ollama-cache
        uses: actions/cache@v4
        with:
          path: ~/.ollama-models
          key: ollama-${{ env.GEN_MODEL }}-${{ env.JUDGE_MODEL }}-${{ env.EMBED_MODEL }}-v1

      - name: Start Ollama container
        run: |
          mkdir -p "$HOME/.ollama-models"
          # Pin this to a tested tag in your own repo instead of latest
          docker run -d --name ollama \
            -p 11434:11434 \
            -v "$HOME/.ollama-models:/root/.ollama" \
            ollama/ollama:latest

          echo "Waiting for Ollama to become ready..."
          for i in $(seq 1 30); do
            if curl -sf "$OLLAMA_HOST/api/version" > /dev/null; then
              echo "Ollama is up"
              exit 0
            fi
            sleep 2
          done
          echo "Ollama failed to start" >&2
          docker logs ollama
          exit 1

      - name: Pull models (cache miss only)
        if: steps.ollama-cache.outputs.cache-hit != 'true'
        run: |
          docker exec ollama ollama pull "$GEN_MODEL"
          docker exec ollama ollama pull "$JUDGE_MODEL"
          docker exec ollama ollama pull "$EMBED_MODEL"

      - name: Warm up models
        run: |
          curl -s "$OLLAMA_HOST/api/generate" \
            -d "{\"model\": \"$JUDGE_MODEL\", \"prompt\": \"ok\", \"stream\": false}" \
            > /dev/null

      - name: Run RAG evaluation
        run: python -m evals.run_eval

      - name: Upload evaluation report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: rag-eval-report
          path: eval-report/

      - name: Comment on pull request
        if: >-
          always() &&
          github.event_name == 'pull_request' &&
          github.event.pull_request.head.repo.full_name == github.repository
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const path = 'eval-report/report.md';
            if (!fs.existsSync(path)) return;
            await github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: fs.readFileSync(path, 'utf8'),
            });

Let's walk through the choices that make this production-grade rather than a demo.

Model caching

Pulling a 4 to 5 GB judge model on every pull request would burn minutes and bandwidth. The actions/cache step persists the Ollama model directory between runs, and the pull step only executes on a cache miss. Bump the v1 suffix in the key whenever you change models on purpose.

Health checks instead of sleep

A fixed sleep 10 is a flaky CI classic. The loop polls /api/version until the server answers, and dumps container logs if it never does. That turns a mysterious failure into a readable one.

Warm-up call

The first request to a freshly started Ollama has to load the model into memory, which can take long enough to trigger Ragas timeouts. A single throwaway call up front avoids blaming your pull request for a cold start.

Scoped triggers and concurrency

The paths filter means documentation-only or frontend-only pull requests do not pay for a 10-minute evaluation. The concurrency block cancels an in-flight run when a developer pushes again, saving runner minutes.

Safe permissions

The workflow requests only read access to contents and write access to pull requests. The comment step is skipped for fork pull requests, where the token is read-only. Those runs still publish the report to the job summary.

Step 6: Turn It into a Real Merge Block

A red CI job does not block anything by itself. You have to tell GitHub that this check is mandatory:

1. Open Settings → Branches (or Rulesets) for your repository.
2. Add a rule for your default branch.
3. Enable Require status checks to pass before merging.
4. Select the rag-eval job as a required check.
5. Optionally enable Require branches to be up to date before merging, so the evaluation always runs against the latest base.

From now on, a pull request that pushes faithfulness below your threshold cannot be merged until someone fixes the problem, or deliberately updates the thresholds in a reviewed change.

Managing the baseline

The baseline is just evals/baseline.json, a small file of scores from the last accepted state:

{
  "faithfulness": 0.91,
  "answer_relevancy": 0.86,
  "context_recall": 0.78,
  "retrieval_hit_rate": 0.93
}

Because it is committed to the repository, raising or lowering the bar is always a visible, reviewable diff. After an intentional improvement lands, copy the numbers from eval-report/scores.json into baseline.json in a follow-up commit. This "ratchet" approach means quality can only move forward unless someone explicitly argues otherwise.

A Real-World Example: Catching a Silent Regression

Let's walk through a scenario that happens all the time. A developer wants shorter answers, so they edit the system prompt and, while cleaning up, delete the line telling the model to answer only from the context. The unit tests pass. The code review looks harmless.

The pull request evaluation comes back like this (illustrative numbers):

| Metric              | Score | Baseline | Delta |
| ------------------- | ----- | -------- | ----- |
| faithfulness        | 0.72  | 0.91     | -0.19 |
| answer_relevancy    | 0.88  | 0.86     | +0.02 |
| context_recall      | 0.78  | 0.78     | +0.00 |
| retrieval_hit_rate  | 0.93  | 0.93     | +0.00 |

Result: FAILED
- faithfulness is 0.72, below the floor of 0.80
- faithfulness regressed by 0.19 vs baseline (0.91 to 0.72)

Read the pattern: retrieval metrics are untouched, answer relevancy is fine, and only faithfulness collapsed. That tells the reviewer immediately that retrieval is healthy and the generation step is inventing content. The developer opens per_question.csv, finds the weekend phone support question now answered with fabricated hours, restores the grounding instruction, and the check goes green.

That diagnostic value is the real payoff. Separate metrics point at separate stages of the pipeline, so you debug in minutes instead of guessing for days.

Handling Speed, Cost, and Flakiness on CPU Runners

Standard GitHub-hosted runners have no GPU, and their CPU and memory specs change over time, so check the current numbers for your plan before choosing models. Here is how to keep things practical:

• Right-size the models. A 3B generator and a 7B to 8B judge is a reasonable starting point. If the judge runs out of memory, drop to a smaller quantized variant or switch to a larger runner.
• Use larger or self-hosted runners if you need a bigger judge. A GPU runner can cut evaluation time dramatically.
• Split fast and slow checks. Run the deterministic retrieval hit rate on every commit and the full Ragas suite on pull requests that touch the pipeline.
• Run a nightly full evaluation against the main branch to catch drift caused by model or dependency updates outside any single pull request.
• Watch the wall-clock time. If evaluation creeps past 20 minutes, developers will start ignoring it. Trim the dataset or upgrade hardware before that happens.

Best Practices

• Pin everything. Model tags, the Ollama image tag, Ragas, LangChain, and Python versions all belong in version control. Unpinned dependencies are the top cause of "it failed but nothing changed."
• Keep the judge separate from the generator. Using the same model to write and grade answers inflates scores. Prefer a different, ideally stronger, judge.
• Never compare across judges. Scores from a 7B judge and a hosted frontier model are not interchangeable. If you change the judge, re-baseline.
• Gate on aggregates, review individuals. Fail the build on averages, but publish per-question scores so humans can inspect the outliers.
• Treat the golden dataset as production code. Require review, keep history, and never delete a failing row just to turn the check green.
• Layer your metrics. Combine deterministic checks with LLM-judged ones. Cheap checks fail fast, and expensive ones give nuance.
• Test the refusal path. Include out-of-scope questions so you know your system says "I don't know" instead of improvising.
• Sample real traffic. Periodically add anonymized production questions to keep the dataset representative of what users actually ask.

Common Mistakes to Avoid

1. Averaging over NaN without tracking it. A broken judge quietly makes everything look fine. Always measure and cap the unscored rate.
2. Setting thresholds by guesswork. Run the evaluation several times on your main branch first, look at the natural variance, and set floors just below that range.
3. Golden data that mirrors the training prompt. If reference answers are copied from the same documents in the same wording, you are testing memorization, not understanding.
4. Letting the dataset go stale. When documentation changes, the golden answers must change with it, or you will fail correct behavior.
5. Ignoring the context window. If Ollama's context length is smaller than your prompts, truncation silently damages both answers and judgments.
6. Chasing a perfect score. A faithfulness of 1.0 across the board usually means a dataset that is too easy. The goal is detecting change, not winning a benchmark.
7. Running the whole suite on every tiny commit. Slow gates get bypassed. Scope triggers with path filters and split fast from slow checks.
8. Treating the judge as ground truth. Small judges make mistakes. Spot-check a handful of scored rows by hand every so often to make sure the metrics still mean what you think they mean.

🚀 Pro Tips

• Add a retrieval-only job first. It needs just the embedding model, finishes in seconds, and already catches a large share of chunking and indexing regressions.
• Publish a trend, not just a snapshot. Store scores.json as a build artifact for each main-branch run and chart it weekly. Slow drift is easier to see on a line graph than in a single report.
• Use matrix builds to compare configurations. Run the same evaluation across two chunk sizes or two embedding models in parallel and let the numbers settle the debate.
• Cache the Python environment too. Combining pip caching with model caching brings warm runs down to a fraction of cold-start time.
• Flag borderline results instead of hard-failing. For scores inside a narrow band above the floor, post a warning comment and require a human approval rather than an automatic block.
• Log the retrieved chunks in your reports. When a pull request fails, being able to see exactly what the model was shown saves enormous debugging time.
• Version your prompts as files. Moving the system prompt into its own file lets your paths filter trigger the evaluation whenever someone edits it, which is precisely when you want it to run.
• Re-baseline deliberately. Whenever you upgrade the judge or Ragas, run the evaluation on main, commit the new baseline in its own pull request, and note why in the description.

📌 Key Takeaways

• RAG quality degrades silently, and only automated evaluation on every relevant change will catch it before users do.
• Faithfulness guards against hallucinations, answer relevancy guards against off-topic replies, and context recall guards against retrieval drift.
• Ollama in a Docker container gives you a free, private, and reproducible judge model inside GitHub Actions.
• A good gate uses three layers: absolute floors, a tolerance against a committed baseline, and a NaN budget for the judge itself.
• Add a deterministic retrieval hit rate to get a fast, LLM-free signal alongside the slower judged metrics.
• The golden dataset is the most valuable asset in the system, so version it, review it, and grow it from real incidents.
• Mark the evaluation job as a required status check, otherwise it is only a suggestion.

Conclusion

Shipping a RAG feature without regression tests is like deploying a payments service with no integration tests. It works until the day it quietly does not, and by then the damage is already public. The good news is that you do not need expensive tooling or a hosted judge to fix it. A containerized Ollama instance, a thoughtful golden dataset, and the Ragas framework give you a private, repeatable quality gate that runs directly in the pull request workflow your team already uses.

Start small. Write 30 good questions, wire up the retrieval hit rate, add the three Ragas metrics, and set conservative thresholds from a few baseline runs. Then let real incidents drive the dataset forward. Within a few weeks, your team will stop debating whether a prompt tweak "feels better" and start looking at numbers, and that shift from opinion to evidence is what separates an AI demo from an AI product you can actually maintain.

References

• Ragas Documentation - official guides for metrics, datasets, and evaluation
• Ragas: Automated Evaluation of Retrieval Augmented Generation (arXiv:2309.15217) - the original research paper behind the framework
• Ollama - model library and documentation
• Ollama on GitHub - source code, API reference, and Docker usage
• Ollama Python Library - the client used in this tutorial
• GitHub Actions Documentation - workflows, caching, concurrency, and permissions
• About protected branches and required status checks - how to make the evaluation job mandatory
• LangChain Ollama Integration - the wrapper classes used to connect Ragas to Ollama