Skip to main content

Multi-node clusters

Rent several machines that sit on the same RDMA fabric and run one distributed job across all of them. A single pod caps you at the GPUs in one box; a cluster lets a training run span many, with the gradients travelling over the fabric instead of the public internet.

When you need one​

A cluster only helps when the job itself spans machines — data-parallel training whose model or batch no longer fits one host, tensor or pipeline parallelism, or a long run you want to finish sooner. Anything that fits inside one machine is cheaper and simpler as an ordinary pod: rent a single 8-GPU host from Browse Pods and skip this page.

Find a cluster​

  1. Open Browse Pods on lium.io and switch to the Clusters tab.
  2. Narrow with the GPU type and GPU count filters if you have a specific card in mind.
  3. Each card is one fabric: the GPU model and per-node GPU count in the header, a badge saying which wire links it, the location, the price and the rent button on the right, and one row per machine with its CPU, memory, VRAM, disk and network.

A card appears when two or more machines on one fabric are free to rent right now. The rows list those machines — the fabric itself can be larger, because its other machines are already rented. A fabric with only one free machine shows no card; that machine is still on sale in the List tab as an ordinary pod.

The location is on the card, not on the rows: one fabric is one switch, so its machines are in one rack. The location comes from the address the provider registered, so read it as the region, not as the street.

InfiniBand or RoCE​

The badge on each cluster card, and on a node in the List tab, says which wire you are buying. Everything else on this page is the same for both.

  • InfiniBand — its own switch and subnet manager. Both nodes report the same subnet manager, so the fabric between them is a measured fact.
  • RoCE — RDMA over Ethernet. Same rental, same overlay, same speed on paper. The grouping is weaker: one provider plus one IPv4 segment, which is usually a shared switch. A switch without lossless queueing (PFC/ECN) carries no RDMA and NCCL drops to TCP. Lium therefore measures the link before it offers the pair, so every RoCE cluster on the tab is one we pushed RDMA traffic across. A pair that fails the measurement is not offered at all.

What a cluster is, exactly​

Every cluster Lium sells is homogeneous and whole:

  • One GPU model, one GPU count per node. A fabric carrying both H100 and H200 machines is offered as two separate cards. Mixed hardware would pace every collective by the slowest card, and an uneven number of ranks per node breaks tensor and pipeline parallel sharding.
  • Whole machines only. You get every GPU on each host. The RDMA devices are handed to a pod only when it holds the entire machine, so a cluster can never share a host with another tenant.
  • All or nothing. Every node you select deploys together, or none does. If one node fails to start, the whole order is rolled back and you are not charged for the rest.

The price shown in the header is the whole order per hour, and it follows your row selection — deselect a machine and the price drops.

What you get when it starts​

Each node comes up as a normal pod with its own SSH access, plus three things an ordinary pod does not have.

A private overlay network. Every node gets an address on 10.42.0.0/24 — node 0 is 10.42.0.1, node 1 is 10.42.0.2, and so on — carried by a WireGuard mesh that exists only between your own pods. NCCL needs one routable address per rank to bootstrap its ring; on the default Docker bridge every container believes it is 172.17.0.2 and a rank ends up dialling itself.

The RDMA devices. /dev/infiniband/ holds rdma_cm and one uverbs* entry per port — the same path for both wires, since RoCE speaks the same verbs API — so NCCL can use the fabric directly, with GPUDirect RDMA where the hardware supports it.

On RoCE the pod also sets NCCL_IB_HCA and NCCL_IB_GID_INDEX, because such a host answers verbs on every Ethernet card it has, including the one carrying the internet, and only one GID of the right card reaches the peer. A wrong pick does not fail, it halves your bandwidth. SSH sessions and nested containers get both, so env | grep NCCL_IB shows them wherever you run.

A shared login for the group. One key pair is minted for the rental and installed on every node, so ssh 10.42.0.2 from any member lands on that peer as root with nothing to set up and no host-key prompt. That is the piece mpirun and DeepSpeed's pdsh launcher were missing; neither tool ships in the image, so install the one you want before you launch with it.

Check all three from inside any node:

ip -4 addr show wg0          # 10.42.0.1/24 on the first node
wg show                      # one peer per other node, with a recent handshake
ls /dev/infiniband/          # rdma_cm  uverbs0 … uverbs7
ping -c3 10.42.0.2           # reach a peer over the overlay
ssh 10.42.0.2 hostname       # log in to a peer as root

If wg show lists no peer, or the ping fails, the fabric did not come up — see Troubleshooting.

Run a job across the nodes​

The launcher has to run on every node at once. With torchrun, each node gets the same command with its own --node_rank, and every node points at node 0's overlay address as the rendezvous host:

# on node 0 (the pod whose wg0 is 10.42.0.1)
torchrun --nnodes=2 --node_rank=0 --nproc_per_node=8 \
  --master_addr=10.42.0.1 --master_port=29500 train.py
# on node 1
torchrun --nnodes=2 --node_rank=1 --nproc_per_node=8 \
  --master_addr=10.42.0.1 --master_port=29500 train.py

Start them in either order but keep both running: torchrun prints nothing while it waits for the other ranks, so a single side on its own looks like it has hung.

--master_addr must be the overlay address, never the pod's public IP. Use a fresh --master_port for each run — a port left behind by a previous attempt makes the next one hang.

Confirm the traffic is on the fabric​

Set NCCL_DEBUG=INFO on the first run and look for the transport NCCL picked:

NCCL INFO NET/IB : Using [0]mlx5_0:1/IB … [7]mlx5_7:1/IB [RO]; OOB wg0:10.42.0.1<0>
NCCL INFO Channel 00/0 : 0[0] -> 1[0] [send] via NET/IB/4/GDRDMA

NET/IB is the fabric. A RoCE cluster prints the same transport with the card named differently:

NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE … ; OOB wg0:10.42.0.1<0>

If you see NET/Socket instead, NCCL fell back to TCP over the overlay and your job will run at a small fraction of the speed it should.

Bring your own image​

At the top level a cluster runs the Lium cluster image. Its entrypoint is what raises the overlay from the injected configuration and exports NCCL_SOCKET_IFNAME=wg0; on any other image the configuration would arrive and nothing would read it, so the rental would report success while NCCL quietly failed.

Your own image runs inside it. The pod ships a Docker daemon whose default runtime injects the RDMA devices into every container it starts, so a nested container gets the fabric with no extra flags:

docker run --rm --gpus all --network host \
  your-registry/your-image:tag  your-command

Do not omit --network host: it puts the container in the pod's network namespace, so it can see wg0 and reach the other nodes. The same default runtime copies the overlay's settings, including NCCL_SOCKET_IFNAME=wg0 and, on a RoCE cluster, the card and GID it picked, into every container that shares that namespace, so you do not pass them yourself. A variable you set with -e is left as you set it.

Current limits​

  • Whole hosts only, so a cluster cannot be assembled from partially free machines.
  • One GPU model at one density per cluster — see above for why.
  • Two nodes minimum. One machine is an ordinary pod with no fabric, and the order is refused.
  • The overlay needs UDP port 51820 between the machines. On a RoCE cluster the nodes reach each other over the fabric they already share, so nothing is open to the internet. On InfiniBand they use their public addresses. A provider who keeps that port closed does not appear in the Clusters tab.
  • A RoCE fabric is inferred from the provider's network. One provider, one IPv4 segment, one address per host. That says the cards are neighbours, not that the switch between them carries RDMA. InfiniBand has no such gap: its subnet manager answers for the fabric.
  • A RoCE fabric is measured before it is offered. While its nodes are free, Lium pushes RDMA traffic between every pair of them over the same card and GID your job will use. A pair the measurement does not cover is dropped from the listing, so an unverified RoCE cluster never reaches this tab.

Troubleshooting​

NET/Socket instead of NET/IB. NCCL did not find the fabric. Check ls /dev/infiniband/ inside the container that runs the job — if it is empty in a nested container, the container was started with a runtime override that dropped the devices; drop the --runtime flag and let the default apply. On RoCE with the devices present, suspect the switch: without lossless queueing it carries no RDMA and NCCL has nowhere to go but TCP. Nothing inside your pod fixes that, so tell support which cluster it was.

The job just sits there. Every rank must be running. torchrun waits silently for the missing ones, which looks identical to a hang. Confirm the process is alive on every node, and that all of them use the same --master_addr, --master_port and --nnodes.

ping 10.42.0.2 gets no answer. The overlay is down. Run wg show on both nodes. For each peer it prints an endpoint address, and it says whether the two nodes ever shook hands. No handshake means UDP port 51820 does not get through to that endpoint address.

A node exits right after it starts. A cluster pod with no active RDMA port refuses to run rather than train over TCP at cluster price. That is the machine, not your image: report the node and rent another fabric.

A cluster disappeared from the Clusters tab. It dropped below two free machines: one of them was rented, or stopped being entirely free. The tab only lists fabrics you can order right now.