Mini-RAG pipeline

A small retrieval pipeline that answers questions from a folder of documents, quotes its sources and checks its own work.

Written to a strict brief: run from a clean checkout, accept a different set of documents, and keep retrieval, ranking and evaluation in code instead of handing them to a language model. It uses no language model at all. Every answer is made of quoted sentences, each cited back to the chunk it came from, and a small HTTP API takes questions.

9stages, always in this order

What it writes

answers.json

{
  "query_id": "Q1",
  "answer_label": "grounded_answer",
  "answer": "Bank withdrawals may take 1 to 3 business days after approval. [Cash withdrawal processing §chunk_2] Withdrawal requests submitted before 17:00 UTC are usually reviewed on the same business day. [Cash withdrawal processing §chunk_1]",
  "citations": [
    "[Cash withdrawal processing §chunk_2]",
    "[Cash withdrawal processing §chunk_1]"
  ]
}

Every sentence in an answer is quoted from a document and cited back to its chunk.

eval.json

"summary": {
  "top3_hit_rate": 1.0,
  "total_queries": 5,
  "hits": 5,
  "partial_hits": 0,
  "misses": 0,
  "mrr_top3": 1.0
}

From the repository's own sample run: four short documents and five questions. A check that it works, not a benchmark.

How it works

Nine stages run in a fixed order, from loading documents to finalising results. Skipping or repeating one raises an error and stops the run.
Documents are split into sentences without breaking on times like 17:00 or abbreviations like e.g., and renaming a file changes nothing.
Retrieval is a hand-written BM25 ranking. A test fails if any sample title, question or answer appears in the code.
Each answer is labelled grounded, not enough context, or conflicting. A claim only counts if the chunk it cites supports it.
Every file it writes is validated, and a second run on the same input is byte-identical. 45 tests cover it.