Skip to content
Skip to this exploration
All explorations

Practical self-hosting · intermediate

21 · The hidden cost of a long conversation

Understand context, output and concurrent memory pressure.

10 min 5 sectionsv1

What you can take away

  • Design a controlled context-length experiment.
  • Write the experiment before running it: fixed inputs, two context conditions, maximum request count, stop threshold, and measurements

Your trail through this idea

Checking saved progress…

A short question works, but the same model struggles after a long conversation. The model has not necessarily changed. The amount of material it must retain and process has.

Prerequisite: complete the preceding lesson or bring its worksheet.

Mark sections complete when you choose. Reading alone does not mark them.

Understand the system

Ollama’s context-length guide describes the model’s accessible token context and the memory implications of larger settings. When reviewed on 14 November 2026, the context guide and FAQ described different defaults, a useful warning against memorizing a number without checking runtime version and configuration. Context includes more than the latest user sentence: system instructions, history, retrieved passages, and output allocation also matter.

Model weights are not the entire memory budget. Runtime buffers, cached attention state, and concurrent requests contribute. Exact costs depend on architecture and implementation; do not use a generic formula as a hardware guarantee.

Checking saved progress…

Work through the case

Original experiment: keep the model and output cap fixed, compare a short synthetic prompt with a longer one containing repeated harmless text, and record latency plus memory observations. Then change concurrency separately. Changing both prompt length and concurrent request count at once would obscure which factor caused the difference.

Checking saved progress…

Make something useful

Try this

Original exercise

Write the experiment before running it: fixed inputs, two context conditions, maximum request count, stop threshold, and measurements. Use a test endpoint and synthetic text only. Start with one request; stop at unexpected memory pressure or service interference. Revert the test configuration afterward. Never use private family documents merely to fill a context window.

Checking saved progress…

Check your reasoning

A quick check · for your own learning

How do you isolate context length in an experiment?

Choose one answer

A self-study prompt. You can retry; this is not an exam.

Checking saved progress…

Keep the result and follow the sources

Success is a comparison with controlled variables and a stated limit. A long supported context is a capacity claim; acceptable latency and quality still need their own evidence.

Sources and scope

  • Primary documentation or archival source, reviewed 14 November 2026. Additional direct links appear beside the claims they support.
  • Editorial basis: Ian's historical self-hosting questions and current repository inspection informed the case selection. Historical requests do not establish current deployment. Private chat records, credentials and host details are excluded.
  • Examples and exercises are original educational synthesis. External sources support the stated technical or historical context, not every hypothetical scenario.
Checking saved progress…

These checks are for reflection and practice. Your answers and reflection notes are not sent anywhere.

Back to all explorations