Practical self-hosting · intermediate
21 · The hidden cost of a long conversation
Understand context, output and concurrent memory pressure.
What you can take away
- Design a controlled context-length experiment.
- Write the experiment before running it: fixed inputs, two context conditions, maximum request count, stop threshold, and measurements
Your trail through this idea
Checking saved progress…
A short question works, but the same model struggles after a long conversation. The model has not necessarily changed. The amount of material it must retain and process has.
Prerequisite: complete the preceding lesson or bring its worksheet.
Understand the system
Ollama’s context-length guide describes the model’s accessible token context and the memory implications of larger settings. When reviewed on 14 November 2026, the context guide and FAQ described different defaults, a useful warning against memorizing a number without checking runtime version and configuration. Context includes more than the latest user sentence: system instructions, history, retrieved passages, and output allocation also matter.
Model weights are not the entire memory budget. Runtime buffers, cached attention state, and concurrent requests contribute. Exact costs depend on architecture and implementation; do not use a generic formula as a hardware guarantee.
Work through the case
Original experiment: keep the model and output cap fixed, compare a short synthetic prompt with a longer one containing repeated harmless text, and record latency plus memory observations. Then change concurrency separately. Changing both prompt length and concurrent request count at once would obscure which factor caused the difference.
Make something useful
Try this
Original exercise
Write the experiment before running it: fixed inputs, two context conditions, maximum request count, stop threshold, and measurements. Use a test endpoint and synthetic text only. Start with one request; stop at unexpected memory pressure or service interference. Revert the test configuration afterward. Never use private family documents merely to fill a context window.
Check your reasoning
A quick check · for your own learning
How do you isolate context length in an experiment?
A self-study prompt. You can retry; this is not an exam.
Keep the result and follow the sources
Success is a comparison with controlled variables and a stated limit. A long supported context is a capacity claim; acceptable latency and quality still need their own evidence.
Sources and scope
- Primary documentation or archival source, reviewed 14 November 2026. Additional direct links appear beside the claims they support.
- Editorial basis: Ian's historical self-hosting questions and current repository inspection informed the case selection. Historical requests do not establish current deployment. Private chat records, credentials and host details are excluded.
- Examples and exercises are original educational synthesis. External sources support the stated technical or historical context, not every hypothetical scenario.