--- sidebar_position: 3.5 --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.lium.io/llms.txt > Use this file to discover all available pages before exploring further. # Before you rent 8 GPUs Two things a multi-GPU listing does not tell you today, and how to find them out in the first two minutes of a rental — before a model download or a tensor-parallel launch burns an hour of an 8-GPU bill. 1. **How the GPUs are wired together.** Eight cards on NVLink and eight cards on PCIe with no peer-to-peer are both "8× H200". Tensor parallelism, FSDP and every NCCL all-reduce run several times slower on the second one, and some launchers do not start on it at all. 2. **How fast the machine pulls from the internet.** The same 756 GB checkpoint took 12 minutes on one 8-GPU node and 75 minutes on another, with identical download settings. The **Download** figure in Browse Pods and `lium ls` does not predict this. Both are being added to the listing itself (an interconnect field, a CDN-measured ingress and egress figure, and matching filters). Until they are there, the checks below are how you know. ## Why the interconnect matters Serving one large model across several GPUs (tensor parallel, `--tensor-parallel-size 8` in vLLM or SGLang) and training with FSDP/ZeRO both synchronise the GPUs after every layer. That traffic goes over NVLink when the board has it (hundreds of GB/s per GPU on HGX H100/H200/B200 boards) and over PCIe otherwise (tens of GB/s at best, shared with the host). A third case is worse than PCIe: a virtualised host that exposes eight GPUs but disables **peer-to-peer** (P2P) between them. NCCL then cannot use GPU-to-GPU copies or shared memory and fails on the first collective with `unhandled cuda error` / `Cuda failure 'operation not supported'`. It only runs with P2P and shared memory disabled — over TCP sockets on the loopback interface. We measured a 1 GB all-reduce across 8 GPUs at 3.1 s on such a node; on an NVLink board it is a few milliseconds. Workloads that do **not** care: one independent process per GPU (batch inference, best-of-N image or video generation, hyper-parameter sweeps, data processing). Those run at full speed on any eight cards. ## First command on a new multi-GPU pod ```bash nvidia-smi topo -m ``` The matrix shows, for every pair of GPUs, what connects them: | Cell | Meaning | Good for TP / FSDP? | |------|---------|---------------------| | `NV#` (e.g. `NV18`, `NV12`, `NV4`) | A bonded set of `#` NVLinks between the two GPUs | Yes — this is what an HGX board looks like (`NV18` on H100/H200, `NV12` on A100) | | `PIX` / `PXB` | PCIe through one or more switches, same host bridge | Workable; several times slower than NVLink | | `PHB` | PCIe through the CPU's host bridge | Slow | | `NODE` / `SYS` | PCIe plus the inter-socket link (QPI/UPI) | Slow; on some hosts P2P is off entirely | | `X` | The GPU itself | — | A healthy 8× H200 host shows `NV18` in every off-diagonal GPU cell. A PCIe box shows `SYS`, `PHB` or `PIX` everywhere and no `NV` at all. Then check whether peer-to-peer copies are allowed at all: ```bash nvidia-smi topo -p2p r # OK = P2P reads work, NS = not supported nvidia-smi nvlink -s # per-link NVLink status; empty or "not supported" on PCIe cards ``` `NS` in every cell of the `-p2p r` matrix means the node is the third case above: NCCL will not work with its defaults. From the CLI, without opening a shell: ```bash lium exec "nvidia-smi topo -m && nvidia-smi topo -p2p r" ``` ### If it is not what you paid for - **Delete the pod and pick another node.** Checking takes under a minute; there is no reason to keep an 8-GPU node that cannot run your job. - **If you must run on it**, tell NCCL to stay off the missing paths. This makes collectives work at socket speed — fine for a quick test, not for serving a 100B+ model with TP=8: ```bash export NCCL_P2P_DISABLE=1 NCCL_SHM_DISABLE=1 # on a virtualised host that also lacks RDMA NICs: export NCCL_IB_DISABLE=1 NCCL_SOCKET_IFNAME=lo ``` - **Restructure the job** so each GPU works alone: pipeline parallel where the per-stage traffic is small, or one replica per GPU for models that fit on a single card. ## Expected download rates What the same 756 GB checkpoint (`hf download` with `hf_xet`, `HF_HOME` on the fast local disk) did on three 8-GPU nodes in one week: | Node | Sustained rate | Time for 756 GB | |------|----------------|-----------------| | 8× H200, US | 2.6–4 GB/s | ~4 min | | 8× B200, US | 1.04 GB/s | 12 min | | 8× H200, other region | 45 MB/s for the first 17 min, then ~200 MB/s | 75 min | That is a 60–90× spread on an identical command. On the slow node PyPI ran at ~1 MB/s as well, so even `pip install` was the bottleneck. At $26/h for the node, the 75 minutes of idle GPUs cost more than the download itself. ### How to read the Download column The **Download** and **Upload** figures in Browse Pods and `lium ls` (Mbps) are smoothed averages of the validator's periodic checks. **Download** is the rate the node achieved fetching a real object of known size and hash during the VerifyX check. **Upload** comes from a speed-test server. They are used to flag nodes under 100 Mbps as slow and to estimate image-pull time. They are **not** a measurement of the path to Hugging Face, PyPI or your object store, and in practice they compress the real spread: nodes listed in the same few-hundred-Mbps band have delivered anything from 45 MB/s to over 1 GB/s from Hugging Face. Divide Mbps by 8 for MB/s. ### Measure before you commit Right after `nvidia-smi topo -m`, spend thirty seconds on the link. Use the disk you will actually download to (`/workspace` or your volume, not the encrypted `/root`). ```bash # a real Hugging Face object, ~550 MB (pip install -U huggingface_hub if `hf` is missing) cd /workspace && time hf download openai-community/gpt2 model.safetensors --local-dir /workspace/probe # a 50 MB CDN object (Cloudflare refuses >= 100 MB with HTTP 403); prints the HTTP code and bytes per second curl -o /dev/null -sS -w 'ingress HTTP %{http_code} %{speed_download} B/s\n' 'https://speed.cloudflare.com/__down?bytes=50000000' # egress, 100 MB head -c 100000000 /dev/zero | curl -o /dev/null -sS -w 'egress %{speed_upload} B/s\n' -X POST --data-binary @- 'https://speed.cloudflare.com/__up' ``` Do the arithmetic once: at 45 MB/s a 750 GB checkpoint takes 4.6 hours; at 1 GB/s it takes 12.5 minutes. If the number does not fit your budget, delete the pod now. Uplink matters when you pull results out. We have seen 30 KB/s on one node and 0.5 MB/s on another in the same class — eleven hours versus forty minutes for a 1.2 GB package. Push results straight from the pod to Hugging Face or an S3 bucket rather than through a laptop, and `tar` many small files first. ## Coming to the listing Lium is adding an `interconnect` object to every executor (NVLink yes/no, number of NVLink links, P2P yes/no, the `nvidia-smi topo -m` classes) and a CDN-measured ingress/egress figure next to the existing ones, with `nvlink` and minimum-ingress filters in the API and in `lium ls`. Until your CLI shows a **Link** column, run the checks above.
For agents and automation ```bash # rent, wait for RUNNING, then verify wiring and link before doing anything else lium up --gpu H200 --count 8 --ttl 6h --name tp8 --yes --no-ssh lium exec tp8 "nvidia-smi topo -m; nvidia-smi topo -p2p r" lium exec tp8 "curl -o /dev/null -sS -w '%{http_code} %{speed_download}\n' 'https://speed.cloudflare.com/__down?bytes=50000000'" ``` Decision rule for a tensor-parallel job: every off-diagonal GPU cell of `topo -m` must be `NV#` and every off-diagonal cell of `topo -p2p r` must be `OK`; otherwise `lium rm tp8` and pick another node. For a weight-heavy job, require the CDN probe to report at least the rate your deadline needs (bytes to download ÷ acceptable seconds).