Practical self-hosting · intermediate
19 · Eight GPUs are not one giant GPU
Reason about placement using memory, topology and workload support.
What you can take away
- Compare candidate placements without inventing current hardware.
- Draw two candidate placements using fictional devices labeled G1, G2, and G3
Your trail through this idea
Checking saved progress…
A model’s weights seem smaller than the sum of several GPUs’ memory. Why might it still fail? Memory capacity is only one part of placement, and memory on separate devices is not automatically one interchangeable pool.
Prerequisite: complete the preceding lesson or bring its worksheet.
Understand the system
Runtime support, model partitioning, device compatibility, communication paths, and per-device allocations all matter. NVIDIA’s management-interface reference includes topology inspection; nvidia-smi topo -m is a read-only way to inspect supported topology information on an appropriate host. Its output needs interpretation and does not itself predict model throughput.
Use hypothetical hardware for the exercise. The number of devices mentioned in a design conversation does not establish Ian’s current inventory, VRAM, or link arrangement. Record current observations privately before making an actual placement recommendation.
Work through the case
Original comparison: configuration A places a small interactive model on one suitable device and a separate batch workload elsewhere. Configuration B spreads one large model across several devices. B may increase feasible model size, yet its inter-device traffic and scheduling can change latency. Neither design wins without a workload and measurements.
Make something useful
Try this
Original exercise
Draw two candidate placements using fictional devices labeled G1, G2, and G3. For each, list per-device memory assumptions, supported partitioning method, communication requirements, expected queue behavior, and failure mode. Then design a benchmark that holds model, prompt set, and output limit constant. Do not change live device assignments for this lab.
Check your reasoning
A quick check · for your own learning
Does adding VRAM across devices prove a model can run there?
A self-study prompt. You can retry; this is not an exam.
Keep the result and follow the sources
The deliverable is a placement hypothesis with falsification criteria. If the measurement contradicts the drawing, trust the measurement and investigate the assumptions.
Sources and scope
- Primary documentation or archival source, reviewed 14 November 2026. Additional direct links appear beside the claims they support.
- Editorial basis: Ian's historical self-hosting questions and current repository inspection informed the case selection. Historical requests do not establish current deployment. Private chat records, credentials and host details are excluded.
- Examples and exercises are original educational synthesis. External sources support the stated technical or historical context, not every hypothetical scenario.