Practical self-hosting · intermediate
24 · Let the short question through
Explore queueing, admission and cancellation.
What you can take away
- Simulate competing workloads and compare waiting time.
- Draw both timelines and calculate arrival-to-start and arrival-to-finish times
Your trail through this idea
Checking saved progress…
A long batch job arrives just before a short interactive question. Total throughput can look excellent while the person waiting for a small answer has a terrible experience.
Prerequisite: complete the preceding lesson or bring its worksheet.
Understand the system
Ollama’s FAQ describes parallel processing and queue-related configuration. Exact limits depend on runtime version and resources. The broader design problem is admission: how much work may enter, how long it may wait, and what happens when capacity is exhausted. Unlimited acceptance merely moves the failure from the front door to memory pressure or timeouts.
Throughput counts completed work over time. Latency measures delay for a request. Fairness concerns how that delay is distributed. A bounded queue and explicit busy response can make a service more predictable. Cancellation must release the work, not only hide it from the browser.
Work through the case
Original paper simulation: one worker receives a 10-unit batch job at time 0, then three 1-unit interactive jobs at time 1. Under non-preemptive FIFO, compute each waiting time. Now reserve a second worker for interactive jobs. Compare the improvement with the additional resource cost; do not pretend the two-worker result is a free scheduling optimization.
Make something useful
Try this
Original exercise
Draw both timelines and calculate arrival-to-start and arrival-to-finish times. Add a maximum waiting time and cancel one queued job. Mark when capacity becomes available again. State whether your design supports preemption; do not assume it. No live load test is required.
Worked queue solution
With one non-preemptive worker, the batch occupies time 0–10. Interactive jobs arriving at time 1 start at 10, 11 and 12, so their waits are 9, 10 and 11 units; completion times measured from arrival are 10, 11 and 12. With an additional reserved worker, they start at 1, 2 and 3: waits 0, 1 and 2, turnaround times 1, 2 and 3. This comparison spends another worker; it is not evidence that a scheduler alone created capacity. Cancelling a queued job should remove its future work, while cancelling a running job releases capacity only when execution actually stops.
Check your reasoning
A quick check · for your own learning
What can high throughput hide?
A self-study prompt. You can retry; this is not an exam.
Keep the result and follow the sources
The result should show why a throughput number alone cannot describe a family-facing service. A good overload policy tells the user what happened and leaves the system able to recover.
Sources and scope
- Primary documentation or archival source, reviewed 14 November 2026. Additional direct links appear beside the claims they support.
- Editorial basis: Ian's historical self-hosting questions and current repository inspection informed the case selection. Historical requests do not establish current deployment. Private chat records, credentials and host details are excluded.
- Examples and exercises are original educational synthesis. External sources support the stated technical or historical context, not every hypothetical scenario.