Test a RAG assistant before users do

Why one accuracy number hides the bugs

Ask "how accurate is the assistant?" and you get a single score that averages away the failures you care about. An assistant can look strong overall while refusing questions it should answer, citing pages it never used, or fabricating an answer on something the documents do not cover. Those are three different failure modes with three different fixes, and they need separate measurements.

I split the work into five scores: retrieval recall, citation support, answer correctness, refusal correctness, and latency with cost per answer. Retrieval is measured before the model writes anything, because if the right passage never reaches the prompt no amount of prompting will save the answer.

Structuring the 30-question test set

I recommend starting with around thirty questions. This is a proposed method, not a completed study I am reporting. Keep the first set small enough to check every row by hand.

A suggested starting mix:

• 15 answerable questions, including a few that need two pages combined.
• 5 questions the documents cannot answer. The correct behaviour is a refusal or a hedge.
• 5 ambiguous questions, where several documents partly apply and the assistant should say which one it relied on.
• 5 adversarial questions: prompt-injection phrasing, out-of-date facts the documents have since corrected, and questions that sound like the site's topic but are not covered.

Write the questions from real use: support tickets, chat logs, or things people actually type. Questions generated from the documents inherit the documents' own assumptions and make retrieval look better than it is. Record the expected source page and the expected behaviour for every question before you run anything, and keep the set in version control so a fix has to survive it.

The five metrics worth scoring

Retrieval recall@k

For each answerable question, did the passage containing the answer appear in the top k results? Score this before the model answers. A question that misses at recall@8 is a retrieval problem, not a model problem.

Citation support

Every factual claim in the answer should trace to a passage in the prompt. An answer that cites a page for a claim the page does not make fails this check, even when the claim happens to be true.

Answer faithfulness

Is the answer true given the retrieved passages? Watch for the model supplying the missing half of a partly covered question from memory, in the same confident tone.

Refusal correctness

On unanswerable questions, does the assistant refuse or hedge instead of guessing? On answerable ones, does it avoid refusing? Count both directions; over-refusal is a bug too.

Latency and cost per answer

A correct answer that arrives too slowly or costs too much is still a failure in production. Record seconds and provider cost alongside the quality scores.

A copyable scorecard

One row per question, scored by hand on a first pass. Blanks stay blank until someone actually looked; an unfilled cell is part of the record. The CSV download has the same columns with 30 blank rows, one per question.

| Question | Type | Expected source | Retrieved@k | Citation supported | Answer correct | Refused correctly | Latency (s) | Notes |
|---|---|---|---|---|---|---|---|---|
| (one row per question, scored by hand) | | | | | | | | |

question_type: answerable | unanswerable | ambiguous | adversarial
  expected_source: the page that should support the answer
  retrieved: did that page appear in the top-k results?
  citation_supported: does each cited passage contain the claim?
  refused_correctly: hedge or refusal on unanswerable, none on answerable
  latency: seconds, plus provider cost if you track it

(Copy the header plus as many blank rows as you have questions.
Blanks stay blank until a person has actually looked at that cell.)

For teams that track this in code, an illustrative sketch of one row as a Python dataclass:

from dataclasses import dataclass

@dataclass
class EvalRow:
    question: str
    question_type: str        # answerable | unanswerable | ambiguous | adversarial
    expected_source: str      # the page that should support the answer
    retrieved: bool           # was the expected source in the top-k results?
    citation_supported: bool  # does each cited passage contain the claim?
    answer_correct: bool
    refused_correctly: bool
    latency_s: float | None
    notes: str

The dataclass is a sketch to adapt. The retrieved and refused_correctly fields help distinguish retrieval failures from incorrect refusals.

Thresholds: automated signals, agreed before the build

Full LLM-judge loops are expensive, so I prefer cheap automated signals with thresholds agreed before the build. UET GPT is a working example of the approach. After retrieval there, a reranker scores each passage against the question (Cohere's rerank first, with local and word-overlap fallbacks and a five-second timeout before the next takes over). The top score then picks the answer instructions:

• No retrieved passages at all: instruct the assistant to refuse and point to the university's site.
• Top score below 0.4: request a hedged answer, prefixed with "Based on limited information available".
• Top score from 0.4 to below 0.6: instruct the assistant to cite a source by name for every factual claim.
• Top score of 0.6 or higher: the normal answer path.

Below 0.6, a second model (Gemini 2.5 Flash) judges each retrieved passage in batches of four for relevance and drops the ones that do not answer the question. That check is skipped when the top score is already 0.6 or higher, because it costs a model call.

Two things matter about these numbers. First, this is exactly what the code does: an earlier version refused on very low scores, which was wrong, because the fallback rerankers' scores are not calibrated the same way as Cohere's and good passages on one provider can score badly on another. Today only an empty result selects refusal instructions. Second, 0.4 and 0.6 are thresholds I chose for that system after watching it answer, not published standards. Calibrate your own on your own documents, and treat them as a contract to revisit, not constants.

The mechanics are written up in the grounded-RAG post, including the retrieval code.

Where UET GPT stands, plainly

UET GPT is live and public. It is an unofficial student project and is not affiliated with UET Taxila. Answers are written by gpt-oss-20b on Groq, falling back to Gemini 2.5 Flash with the same prompt and the same limits on what it may say.

There is no published accuracy number for it. The repository has an answer-accuracy evaluation script, but I have not run the 30-question protocol on it and published a figure. If I run the protocol as part of a project, the result goes into the scorecard above with its test set attached, so anyone can rerun it.

The broader record of what I have measured and not measured is on the evidence page.

Five ways RAG evaluations mislead

1. Cherry-picked source documents. Evaluating on five clean pages when the index serves two hundred messy ones tells you about the five pages.
2. No unanswerable cases. A suite where everything is answerable cannot see the most damaging failure: a confident made-up answer.
3. Synthetic questions. Questions generated from the documents rewrite themselves into retrievable phrasing. The retrieval recall you measure does not transfer to how users phrase things.
4. Judge bias. The same model family that wrote the answer judges the answer, and both share taste and blind spots. Spot-check judge verdicts by hand and count the disagreements.
5. Testing only the happy path. Score refusal, injection attempts and provider outages. If your test runner never sees the fallback model, neither has your evaluation.

Frequently asked questions