Data & AI Project
RAG Document Q&A
Retrieval-augmented generation answers questions using your documents rather than the model's memory: you chunk the text, embed it, retrieve the passages closest to the question and pass them to the model as context. Almost all of the quality comes from retrieval, not from the model — which is what makes it a real engineering project.
The brief
Ingest a document set, answer questions about it with citations to the source passages, and say "I do not know" when retrieval finds nothing relevant. Evaluate retrieval quality separately from answer quality.
What it demonstrates
That you can build on top of an LLM without treating it as magic — and that you measured retrieval instead of trusting a demo.
What "done" looks like
Build all of these and the project is finished. Anything past that is in the stretch goals.
- Ingest PDFs or Markdown into a searchable index
- Semantic chunking with overlap
- Retrieval returning the top passages for a question
- Answers grounded in those passages, with citations
- An explicit "not found in these documents" path
- Retrieval evaluated separately from answer quality
How to build it
- 1
Ingest and normalise
Extract text, strip boilerplate, keep the source and page for every passage — citations depend on it.
- 2
Chunk on meaning
Split at headings and paragraphs with slight overlap, not at a fixed character count. This decides most of your quality.
- 3
Embed the chunks
Pick one embedding model and keep it consistent — mixing models makes the vector space meaningless.
- 4
Build the index
Start in memory or in SQLite. A dedicated vector database is a later optimisation, not a starting requirement.
- 5
Retrieve and rerank
Fetch more candidates than you need, then rerank. Cheap, and it lifts answer quality noticeably.
- 6
Ground the generation
Pass only the retrieved passages, require citations, and instruct the model to decline when they do not contain the answer.
- 7
Evaluate retrieval on its own
Write question-passage pairs and measure recall@k. If retrieval is wrong, no prompt will save the answer.
- 8
Test that it refuses
Ask questions your documents cannot answer. A system that always answers is hallucinating, not working.
Once it works
Only after the definition of done is met. Half-finished ambition reads worse than a small finished thing.
- Add hybrid search — keyword plus vector — and measure the lift
- Add conversation memory with query rewriting for follow-ups
- Show the retrieved passages in the UI so answers are auditable
Frequently Asked Questions
How should I chunk documents?
Split on semantic boundaries — headings, paragraphs — rather than a fixed character count, and overlap slightly so an answer spanning a boundary is not cut in half. Chunking is the highest-leverage decision in the whole system and the one most people skip past.
Do I need a vector database?
Not at first. For a few thousand chunks, an in-memory index or SQLite with a vector extension is plenty and far easier to debug. Add a dedicated store when scale demands it — knowing when is a better story than adopting one immediately.
How do I stop it hallucinating?
Ground it: instruct the model to answer only from the supplied passages, cite them, and say it does not know when they do not contain the answer. Then verify by asking questions your documents genuinely cannot answer — that test is what separates a real RAG system from a demo.