Skip to content
Skip to this exploration
All explorations

Practical self-hosting · intermediate

24 · Let the short question through

Explore queueing, admission and cancellation.

10 min 5 sectionsv1

What you can take away

  • Simulate competing workloads and compare waiting time.
  • Draw both timelines and calculate arrival-to-start and arrival-to-finish times

Your trail through this idea

Checking saved progress…

A long batch job arrives just before a short interactive question. Total throughput can look excellent while the person waiting for a small answer has a terrible experience.

Prerequisite: complete the preceding lesson or bring its worksheet.

Mark sections complete when you choose. Reading alone does not mark them.

Understand the system

Ollama’s FAQ describes parallel processing and queue-related configuration. Exact limits depend on runtime version and resources. The broader design problem is admission: how much work may enter, how long it may wait, and what happens when capacity is exhausted. Unlimited acceptance merely moves the failure from the front door to memory pressure or timeouts.

Throughput counts completed work over time. Latency measures delay for a request. Fairness concerns how that delay is distributed. A bounded queue and explicit busy response can make a service more predictable. Cancellation must release the work, not only hide it from the browser.

Checking saved progress…

Work through the case

Original paper simulation: one worker receives a 10-unit batch job at time 0, then three 1-unit interactive jobs at time 1. Under non-preemptive FIFO, compute each waiting time. Now reserve a second worker for interactive jobs. Compare the improvement with the additional resource cost; do not pretend the two-worker result is a free scheduling optimization.

Checking saved progress…

Make something useful

Try this

Original exercise

Draw both timelines and calculate arrival-to-start and arrival-to-finish times. Add a maximum waiting time and cancel one queued job. Mark when capacity becomes available again. State whether your design supports preemption; do not assume it. No live load test is required.

Worked queue solution

With one non-preemptive worker, the batch occupies time 0–10. Interactive jobs arriving at time 1 start at 10, 11 and 12, so their waits are 9, 10 and 11 units; completion times measured from arrival are 10, 11 and 12. With an additional reserved worker, they start at 1, 2 and 3: waits 0, 1 and 2, turnaround times 1, 2 and 3. This comparison spends another worker; it is not evidence that a scheduler alone created capacity. Cancelling a queued job should remove its future work, while cancelling a running job releases capacity only when execution actually stops.

Checking saved progress…

Check your reasoning

A quick check · for your own learning

What can high throughput hide?

Choose one answer

A self-study prompt. You can retry; this is not an exam.

Checking saved progress…

Keep the result and follow the sources

The result should show why a throughput number alone cannot describe a family-facing service. A good overload policy tells the user what happened and leaves the system able to recover.

Sources and scope

  • Primary documentation or archival source, reviewed 14 November 2026. Additional direct links appear beside the claims they support.
  • Editorial basis: Ian's historical self-hosting questions and current repository inspection informed the case selection. Historical requests do not establish current deployment. Private chat records, credentials and host details are excluded.
  • Examples and exercises are original educational synthesis. External sources support the stated technical or historical context, not every hypothetical scenario.
Checking saved progress…

These checks are for reflection and practice. Your answers and reflection notes are not sent anywhere.

Back to all explorations