Skip to content
Skip to this exploration
All explorations

Practical self-hosting · intermediate

19 · Eight GPUs are not one giant GPU

Reason about placement using memory, topology and workload support.

10 min 5 sectionsv1

What you can take away

  • Compare candidate placements without inventing current hardware.
  • Draw two candidate placements using fictional devices labeled G1, G2, and G3

Your trail through this idea

Checking saved progress…

A model’s weights seem smaller than the sum of several GPUs’ memory. Why might it still fail? Memory capacity is only one part of placement, and memory on separate devices is not automatically one interchangeable pool.

Prerequisite: complete the preceding lesson or bring its worksheet.

Mark sections complete when you choose. Reading alone does not mark them.

Understand the system

Runtime support, model partitioning, device compatibility, communication paths, and per-device allocations all matter. NVIDIA’s management-interface reference includes topology inspection; nvidia-smi topo -m is a read-only way to inspect supported topology information on an appropriate host. Its output needs interpretation and does not itself predict model throughput.

Use hypothetical hardware for the exercise. The number of devices mentioned in a design conversation does not establish Ian’s current inventory, VRAM, or link arrangement. Record current observations privately before making an actual placement recommendation.

Checking saved progress…

Work through the case

Original comparison: configuration A places a small interactive model on one suitable device and a separate batch workload elsewhere. Configuration B spreads one large model across several devices. B may increase feasible model size, yet its inter-device traffic and scheduling can change latency. Neither design wins without a workload and measurements.

Checking saved progress…

Make something useful

Try this

Original exercise

Draw two candidate placements using fictional devices labeled G1, G2, and G3. For each, list per-device memory assumptions, supported partitioning method, communication requirements, expected queue behavior, and failure mode. Then design a benchmark that holds model, prompt set, and output limit constant. Do not change live device assignments for this lab.

Checking saved progress…

Check your reasoning

A quick check · for your own learning

Does adding VRAM across devices prove a model can run there?

Choose one answer

A self-study prompt. You can retry; this is not an exam.

Checking saved progress…

Keep the result and follow the sources

The deliverable is a placement hypothesis with falsification criteria. If the measurement contradicts the drawing, trust the measurement and investigate the assumptions.

Sources and scope

  • Primary documentation or archival source, reviewed 14 November 2026. Additional direct links appear beside the claims they support.
  • Editorial basis: Ian's historical self-hosting questions and current repository inspection informed the case selection. Historical requests do not establish current deployment. Private chat records, credentials and host details are excluded.
  • Examples and exercises are original educational synthesis. External sources support the stated technical or historical context, not every hypothetical scenario.
Checking saved progress…

These checks are for reflection and practice. Your answers and reflection notes are not sent anywhere.

Back to all explorations