Skip to main content

Before you rent 8 GPUs

Two things a multi-GPU listing does not tell you today, and how to find them out in the first two minutes of a rental — before a model download or a tensor-parallel launch burns an hour of an 8-GPU bill.

  1. How the GPUs are wired together. Eight cards on NVLink and eight cards on PCIe with no peer-to-peer are both "8× H200". Tensor parallelism, FSDP and every NCCL all-reduce run several times slower on the second one, and some launchers do not start on it at all.
  2. How fast the machine pulls from the internet. The same 756 GB checkpoint took 12 minutes on one 8-GPU node and 75 minutes on another, with identical download settings. The Download figure in Browse Pods and lium ls does not predict this.

Both are being added to the listing itself (an interconnect field, a CDN-measured ingress and egress figure, and matching filters). Until they are there, the checks below are how you know.

Why the interconnect matters​

Serving one large model across several GPUs (tensor parallel, --tensor-parallel-size 8 in vLLM or SGLang) and training with FSDP/ZeRO both synchronise the GPUs after every layer. That traffic goes over NVLink when the board has it (hundreds of GB/s per GPU on HGX H100/H200/B200 boards) and over PCIe otherwise (tens of GB/s at best, shared with the host).

A third case is worse than PCIe: a virtualised host that exposes eight GPUs but disables peer-to-peer (P2P) between them. NCCL then cannot use GPU-to-GPU copies or shared memory and fails on the first collective with unhandled cuda error / Cuda failure 'operation not supported'. It only runs with P2P and shared memory disabled — over TCP sockets on the loopback interface. We measured a 1 GB all-reduce across 8 GPUs at 3.1 s on such a node; on an NVLink board it is a few milliseconds.

Workloads that do not care: one independent process per GPU (batch inference, best-of-N image or video generation, hyper-parameter sweeps, data processing). Those run at full speed on any eight cards.

First command on a new multi-GPU pod​

nvidia-smi topo -m

The matrix shows, for every pair of GPUs, what connects them:

CellMeaningGood for TP / FSDP?
NV# (e.g. NV18, NV12, NV4)A bonded set of # NVLinks between the two GPUsYes — this is what an HGX board looks like (NV18 on H100/H200, NV12 on A100)
PIX / PXBPCIe through one or more switches, same host bridgeWorkable; several times slower than NVLink
PHBPCIe through the CPU's host bridgeSlow
NODE / SYSPCIe plus the inter-socket link (QPI/UPI)Slow; on some hosts P2P is off entirely
XThe GPU itself—

A healthy 8× H200 host shows NV18 in every off-diagonal GPU cell. A PCIe box shows SYS, PHB or PIX everywhere and no NV at all.

Then check whether peer-to-peer copies are allowed at all:

nvidia-smi topo -p2p r      # OK = P2P reads work, NS = not supported
nvidia-smi nvlink -s        # per-link NVLink status; empty or "not supported" on PCIe cards

NS in every cell of the -p2p r matrix means the node is the third case above: NCCL will not work with its defaults.

From the CLI, without opening a shell:

lium exec <pod> "nvidia-smi topo -m && nvidia-smi topo -p2p r"

If it is not what you paid for​

  • Delete the pod and pick another node. Checking takes under a minute; there is no reason to keep an 8-GPU node that cannot run your job.

  • If you must run on it, tell NCCL to stay off the missing paths. This makes collectives work at socket speed — fine for a quick test, not for serving a 100B+ model with TP=8:

    export NCCL_P2P_DISABLE=1 NCCL_SHM_DISABLE=1
    # on a virtualised host that also lacks RDMA NICs:
    export NCCL_IB_DISABLE=1 NCCL_SOCKET_IFNAME=lo

  • Restructure the job so each GPU works alone: pipeline parallel where the per-stage traffic is small, or one replica per GPU for models that fit on a single card.

Expected download rates​

What the same 756 GB checkpoint (hf download with hf_xet, HF_HOME on the fast local disk) did on three 8-GPU nodes in one week:

NodeSustained rateTime for 756 GB
8× H200, US2.6–4 GB/s~4 min
8× B200, US1.04 GB/s12 min
8× H200, other region45 MB/s for the first 17 min, then ~200 MB/s75 min

That is a 60–90× spread on an identical command. On the slow node PyPI ran at ~1 MB/s as well, so even pip install was the bottleneck. At $26/h for the node, the 75 minutes of idle GPUs cost more than the download itself.

How to read the Download column​

The Download and Upload figures in Browse Pods and lium ls (Mbps) are smoothed averages of the validator's periodic checks. Download is the rate the node achieved fetching a real object of known size and hash during the VerifyX check. Upload comes from a speed-test server. They are used to flag nodes under 100 Mbps as slow and to estimate image-pull time. They are not a measurement of the path to Hugging Face, PyPI or your object store, and in practice they compress the real spread: nodes listed in the same few-hundred-Mbps band have delivered anything from 45 MB/s to over 1 GB/s from Hugging Face. Divide Mbps by 8 for MB/s.

Measure before you commit​

Right after nvidia-smi topo -m, spend thirty seconds on the link. Use the disk you will actually download to (/workspace or your volume, not the encrypted /root).

# a real Hugging Face object, ~550 MB  (pip install -U huggingface_hub if `hf` is missing)
cd /workspace && time hf download openai-community/gpt2 model.safetensors --local-dir /workspace/probe

# a 50 MB CDN object (Cloudflare refuses >= 100 MB with HTTP 403); prints the HTTP code and bytes per second
curl -o /dev/null -sS -w 'ingress HTTP %{http_code} %{speed_download} B/s\n' 'https://speed.cloudflare.com/__down?bytes=50000000'

# egress, 100 MB
head -c 100000000 /dev/zero | curl -o /dev/null -sS -w 'egress %{speed_upload} B/s\n' -X POST --data-binary @- 'https://speed.cloudflare.com/__up'

Do the arithmetic once: at 45 MB/s a 750 GB checkpoint takes 4.6 hours; at 1 GB/s it takes 12.5 minutes. If the number does not fit your budget, delete the pod now.

Uplink matters when you pull results out. We have seen 30 KB/s on one node and 0.5 MB/s on another in the same class — eleven hours versus forty minutes for a 1.2 GB package. Push results straight from the pod to Hugging Face or an S3 bucket rather than through a laptop, and tar many small files first.

In the listing​

Every executor on GET /executors carries an interconnect object (NVLink yes/no, number of NVLink links, P2P yes/no, the nvidia-smi topo -m classes) and a one-word nvlink verdict, and the API filters on nvlink=true and min_download_mbps (the VerifyX download figure). See Interconnect and download speed on GET /executors. The lium ls Link column is unreleased (lium#149); until your CLI shows it, run the checks above.

For agents and automation
# rent, wait for RUNNING, then verify wiring and link before doing anything else
lium up --gpu H200 --count 8 --ttl 6h --name tp8 --yes --no-ssh
lium exec tp8 "nvidia-smi topo -m; nvidia-smi topo -p2p r"
lium exec tp8 "curl -o /dev/null -sS -w '%{http_code} %{speed_download}\n' 'https://speed.cloudflare.com/__down?bytes=50000000'"

Decision rule for a tensor-parallel job: every off-diagonal GPU cell of topo -m must be NV# and every off-diagonal cell of topo -p2p r must be OK; otherwise lium rm tp8 and pick another node. For a weight-heavy job, require the CDN probe to report at least the rate your deadline needs (bytes to download ÷ acceptable seconds).