Building Grounded RAG Systems: Retrieval and Refusal

Why plain RAG isn't enough

The tutorial version of RAG is three steps: embed your documents, find the nearest few to the question, and paste them into the prompt. It fails in predictable ways:

• Nearest isn't relevant. Vector search can return a nearby passage even when none answers the question. A question about hostel fees can pull back a page about tuition fees, and the model will answer from it confidently.
• The model fills gaps. When the passages cover half the question, the model often supplies the other half from memory, in the same confident tone.
• One provider is a single point of failure. If the only model API is rate-limited, the assistant is down.

None of these are fixed by a better prompt alone. They need decisions made in code, before the model is called.

Step 1: Let the reranker's score choose the answer mode

After retrieval, a reranker scores each passage against the question. UET GPT uses Cohere's rerank-v4.0-fast, with a local bge-reranker-large model and a simple word-overlap score as fallbacks. A slow reranker times out after 5 seconds and the next one takes over.

The top score then picks an instruction that is added to the prompt. This is a shortened version of the real function:

function determineConfidenceTier(results: { relevanceScore: number }[]) {
  if (results.length === 0) {
    return {
      tier: "refuse",
      instruction:
        "No relevant information was found. Do not state any UET-specific facts. Respond with: " +
        "'I don't have verified information about this - please check uettaxila.edu.pk directly.'",
    };
  }
  const topScore = results[0].relevanceScore;

  if (topScore < 0.4) {
    return {
      tier: "hedge",
      instruction:
        "Answer ONLY from the provided context. Prefix your answer with " +
        "'Based on limited information available - ' and end with " +
        "'For authoritative details, please verify at uettaxila.edu.pk.'",
    };
  }
  if (topScore < 0.6) {
    return {
      tier: "cite",
      instruction:
        "Cite specific sources by name for every factual claim. " +
        "If a claim cannot be attributed to a source, leave it out.",
    };
  }
  return { tier: "normal", instruction: "" };
}

An earlier version refused whenever the top score was very low. That turned out to be wrong: scores from the fallback rerankers aren't calibrated the same way as Cohere's, so good passages sometimes scored under 0.2 and the assistant refused questions it could have answered. Now only an empty result selects refusal instructions. A low score selects hedging, and the relevance judge in the next step decides what is actually irrelevant.

Step 2: Ask a second model whether the passages are relevant

Reranker scores rank passages against each other. They don't say whether the best one answers the question. For that, UET GPT runs a corrective-retrieval check (CRAG): Gemini 2.5 Flash looks at the passages in batches of four and marks each one relevant or not. Irrelevant passages are dropped before the answer is written.

The check costs a model call, so it is skipped when the reranker's top score is already 0.6 or higher. Below that, it always runs.

Step 3: Fall back to a second model, not a second guess

Answers are written by gpt-oss-20b on Groq. When Groq fails or is rate-limited, the same prompt, with the same passages and the same instruction, goes to Gemini 2.5 Flash. The fallback changes who writes the answer, not what it is allowed to say.

Every model task (query rewriting, answer writing, the relevance check, embeddings, reranking) is listed in one registry with its primary and fallback model. That gives me one place to update a retired model. A failed call with no working fallback reports an error.

Step 4: Reject prompt injection before retrieval

Questions are checked against a short list of patterns such as "ignore previous instructions", [INST] and fake system: lines before anything else runs. The list was once too broad: a phrase like "you are an" rejected the ordinary question "What documents do I need if you are an overseas applicant?". Each pattern now needs the full hijack phrase. Pattern lists are a first pass, not a complete defence. Retrieved text is also fenced off in the prompt, with instructions to treat web-page text as content. Neither measure guarantees that an injection will be rejected.

Where it still goes wrong

• Stale pages. A wrong answer can start with the crawl: a fee page that changed after the last crawl, or two pages that disagree. Retrieval faithfully finds the wrong fact.
• Hedged answers can still be wrong. "Based on limited information available" is a warning, not a guarantee. A careful reader gets the signal; a hurried one may not.
• No published accuracy number yet. The repository has an answer-accuracy evaluation script, but I haven't published a figure from it.

Evaluating RAG without synthetic inflation

UET GPT has no published accuracy figure. I would evaluate it with questions from real users and expected answers checked against source documents.

The protocol I would use, and would run with a client before calling an assistant finished:

1. 30 questions from real use, not generated from the documents. They should include simple lookups, questions that need two pages combined, and questions the documents cannot answer.
2. Retrieval recall. For each question, did the passage that contains the answer appear in the top results? This is measured before the model writes anything.
3. Citation support. Does every factual claim in the answer trace to a retrieved passage? An answer that cites a page it did not use fails this check.
4. Refusal correctness. On the questions the documents cannot answer, does the assistant refuse or hedge instead of guessing? On answerable questions, does it avoid refusing?
5. Latency and cost per answer. A correct answer that takes too long or costs too much is still a failure in production.

A 30-question set is a method I would use to scope and check the build, not a completed result I am reporting here. The number matters less than the failure list it produces: which questions missed retrieval, which answers cited the wrong page, and which refusals were unnecessary.

If you have a documentation set and want this refusal path built, that is the grounded RAG assistant service. The live example is UET GPT.

Convex or a separate vector database?

UET GPT keeps vectors, documents and chat history in Convex. For an app like this the main gain is one system instead of two: no syncing a vector store with a relational database, and the chat UI updates live through Convex subscriptions. A dedicated vector database makes more sense when the index is much larger than the app's own data or is shared across several apps.

Frequently Asked Questions

Can RAG eliminate hallucinations?

No. It can make them rarer and easier to spot. Retrieval decides what the model sees, and instructions decide what it may claim, but the model can still misread a passage or answer from memory. The useful goals are: refuse when nothing relevant was found, hedge when the evidence is weak, and cite sources so a reader can check.

How do you decide when a RAG assistant should refuse to answer?

Use the retrieval results, not the model's own judgement. In UET GPT, empty retrieval selects refusal instructions. Low reranker scores select hedging instructions, and a separate relevance check removes passages that don't answer the question.

What happens when the main LLM provider is down?

The same prompt and passages go to a fallback model. In UET GPT, answers come from gpt-oss-20b on Groq and fall back to Gemini 2.5 Flash. Because the instruction travels with the prompt, the fallback model has the same limits on what it may say.

Why rerank results after vector search?

Vector similarity finds passages that look like the question. A reranker reads the question and passage together and scores how well one answers the other, which is a better signal for choosing what the model sees and for deciding how confident the answer should be.