Practical self-hosting · intermediate
25 · Build a library that can say it does not know
Understand retrieval as evidence selection before generation.
What you can take away
- Evaluate retrieval on answerable and unanswerable questions.
- Create these three synthetic documents locally
Your trail through this idea
Checking saved progress…
Ian’s linked local-AI tools suggest a useful goal: answer questions from a controlled collection. The hard part is not merely finding similar text. It is returning evidence that actually supports the answer.
Prerequisite: complete the preceding lesson or bring its worksheet.
Understand the system
Retrieval selects candidate passages; generation writes a response using them. Similarity does not establish truth, access rights, or completeness. A passage can mention the right words while answering a different question. Qdrant’s filtering documentation shows how structured conditions can constrain searches alongside vector retrieval. Those conditions must reflect your application’s requirements.
Start with a tiny collection whose answers you know. Keep source identifiers attached through chunking, indexing, retrieval, and rendering. An answer should identify its supporting passage. If no passage supports the requested fact, the appropriate outcome is a clear lack-of-evidence response.
Work through the case
Original corpus: document A says the fictional library opens at 10; B says the fictional radio repeats at noon; C says the fictional archive accepts text files. Ask for the library opening time, then ask for its closing time. Only the first is supported. A plausible closing time is a failure, even if the prose sounds helpful.
Make something useful
Try this
Original exercise
Create these three synthetic documents locally. Build a manual retrieval table first: question, expected passage, expected answer, and unsupported claims to reject. If you later index them in a test service, compare its results to the table. Do not use the private GED as a convenient test corpus. Delete only the disposable collection after the lab.
Runnable evidence-selection lab
Save this original Python 3 example as lesson_retrieval.py and run python3 lesson_retrieval.py. It is a tiny keyword baseline, not vector search or an LLM. It uses only invented documents and writes no data.
docs = {
"A": "The fictional library opens at 10.",
"B": "The fictional radio repeats at noon.",
"C": "The fictional archive accepts text files.",
}
def retrieve(required_words):
return [(key, text) for key, text in docs.items()
if all(word in text.lower() for word in required_words)]
opening = retrieve(["library", "opens"])
closing = retrieve(["library", "closes"])
assert opening == [("A", docs["A"])]
assert closing == []
print("Opening evidence:", opening)
print("Closing evidence:", closing)
The opening query returns A; the closing query returns an empty list. Now change all to any: the closing query will incorrectly retrieve A merely because it mentions the library, and the assertion should fail. This illustrates relevance versus support. Restore all and add a fourth synthetic document containing a closing time; update the expected result only when that new passage actually supports it. Keyword matching remains brittle to synonyms, negation and wording. A vector system must still pass support tests rather than merely finding semantically nearby text. Cleanup is removal of the practice script.
Check your reasoning
A quick check · for your own learning
What should an answer do when the corpus lacks the closing time?
A self-study prompt. You can retry; this is not an exam.
Keep the result and follow the sources
Success means correct evidence selection and a successful no-evidence case. Measure citation support separately from answer fluency; a polished unsupported answer is still wrong.
Sources and scope
- Primary documentation or archival source, reviewed 14 November 2026. Additional direct links appear beside the claims they support.
- Editorial basis: Ian's historical self-hosting questions and current repository inspection informed the case selection. Historical requests do not establish current deployment. Private chat records, credentials and host details are excluded.
- Examples and exercises are original educational synthesis. External sources support the stated technical or historical context, not every hypothetical scenario.