Skip to content
Skip to this exploration
All explorations

Practical self-hosting · intermediate

25 · Build a library that can say it does not know

Understand retrieval as evidence selection before generation.

10 min 5 sectionsv1

What you can take away

  • Evaluate retrieval on answerable and unanswerable questions.
  • Create these three synthetic documents locally

Your trail through this idea

Checking saved progress…

Ian’s linked local-AI tools suggest a useful goal: answer questions from a controlled collection. The hard part is not merely finding similar text. It is returning evidence that actually supports the answer.

Prerequisite: complete the preceding lesson or bring its worksheet.

Mark sections complete when you choose. Reading alone does not mark them.

Understand the system

Retrieval selects candidate passages; generation writes a response using them. Similarity does not establish truth, access rights, or completeness. A passage can mention the right words while answering a different question. Qdrant’s filtering documentation shows how structured conditions can constrain searches alongside vector retrieval. Those conditions must reflect your application’s requirements.

Start with a tiny collection whose answers you know. Keep source identifiers attached through chunking, indexing, retrieval, and rendering. An answer should identify its supporting passage. If no passage supports the requested fact, the appropriate outcome is a clear lack-of-evidence response.

Checking saved progress…

Work through the case

Original corpus: document A says the fictional library opens at 10; B says the fictional radio repeats at noon; C says the fictional archive accepts text files. Ask for the library opening time, then ask for its closing time. Only the first is supported. A plausible closing time is a failure, even if the prose sounds helpful.

Checking saved progress…

Make something useful

Try this

Original exercise

Create these three synthetic documents locally. Build a manual retrieval table first: question, expected passage, expected answer, and unsupported claims to reject. If you later index them in a test service, compare its results to the table. Do not use the private GED as a convenient test corpus. Delete only the disposable collection after the lab.

Runnable evidence-selection lab

Save this original Python 3 example as lesson_retrieval.py and run python3 lesson_retrieval.py. It is a tiny keyword baseline, not vector search or an LLM. It uses only invented documents and writes no data.

docs = {
    "A": "The fictional library opens at 10.",
    "B": "The fictional radio repeats at noon.",
    "C": "The fictional archive accepts text files.",
}
def retrieve(required_words):
    return [(key, text) for key, text in docs.items()
            if all(word in text.lower() for word in required_words)]

opening = retrieve(["library", "opens"])
closing = retrieve(["library", "closes"])
assert opening == [("A", docs["A"])]
assert closing == []
print("Opening evidence:", opening)
print("Closing evidence:", closing)

The opening query returns A; the closing query returns an empty list. Now change all to any: the closing query will incorrectly retrieve A merely because it mentions the library, and the assertion should fail. This illustrates relevance versus support. Restore all and add a fourth synthetic document containing a closing time; update the expected result only when that new passage actually supports it. Keyword matching remains brittle to synonyms, negation and wording. A vector system must still pass support tests rather than merely finding semantically nearby text. Cleanup is removal of the practice script.

Checking saved progress…

Check your reasoning

A quick check · for your own learning

What should an answer do when the corpus lacks the closing time?

Choose one answer

A self-study prompt. You can retry; this is not an exam.

Checking saved progress…

Keep the result and follow the sources

Success means correct evidence selection and a successful no-evidence case. Measure citation support separately from answer fluency; a polished unsupported answer is still wrong.

Sources and scope

  • Primary documentation or archival source, reviewed 14 November 2026. Additional direct links appear beside the claims they support.
  • Editorial basis: Ian's historical self-hosting questions and current repository inspection informed the case selection. Historical requests do not establish current deployment. Private chat records, credentials and host details are excluded.
  • Examples and exercises are original educational synthesis. External sources support the stated technical or historical context, not every hypothetical scenario.
Checking saved progress…

These checks are for reflection and practice. Your answers and reflection notes are not sent anywhere.

Back to all explorations