Skip to main content

Multi-node clusters

Rent several machines that sit on the same RDMA fabric and run one distributed job across all of them. A single pod caps you at the GPUs in one box; a cluster lets a training run span many, with the gradients traveling over the fabric.

When you need one​

A cluster only helps when the job itself spans machines — data-parallel training whose model or batch is too large for one host, tensor or pipeline parallelism, or a long run you want to finish sooner. Anything that fits inside one machine is cheaper and simpler as an ordinary pod: rent a single 8-GPU host from Browse Pods and skip this page.

Find a cluster​

  1. Open Browse Pods on lium.io and switch to the Clusters tab.
  2. Narrow with the GPU type and GPU count filters if you have a specific card in mind.
  3. Each card is one fabric: the GPU model and per-node GPU count in the header, a badge saying which wire links it, the location, the price and the rent button on the right, and one row per machine with its CPU, memory, VRAM, disk and network.

A card appears when two or more machines on one fabric are free to rent right now. The rows list those machines — the fabric itself can be larger, because its other machines are already rented. A fabric with only one free machine shows no card; that machine is still on sale in the List tab as an ordinary pod.

The location is on the card, not on the rows: one fabric is one switch, so its machines are in one rack. The location comes from the address the provider registered, so read it as the region, not as the street.

InfiniBand or RoCE​

The badge on each cluster card, and on a node in the List tab, says which wire you are buying. Everything else on this page is the same for both.

  • InfiniBand — its own switch and subnet manager. Both nodes report the same subnet manager, so the fabric between them is a measured fact.
  • RoCE — RDMA over Ethernet. Same rental, same overlay, same speed on paper. The grouping is weaker: one provider plus one IPv4 segment, which is usually a shared switch. A switch without lossless queueing (PFC/ECN) carries no RDMA and NCCL drops to TCP. Lium therefore measures the link before it offers the pair, so every RoCE cluster on the tab is one we pushed RDMA traffic across. A pair that fails the measurement is not offered at all.

What a cluster is, exactly​

Every cluster Lium sells is homogeneous and whole:

  • One GPU model, one GPU count per node. A fabric carrying both H100 and H200 machines is offered as two separate cards. Mixed hardware would pace every collective by the slowest card, and an uneven number of ranks per node breaks tensor and pipeline parallel sharding.
  • Whole machines only. You get every GPU on each host. The RDMA devices are handed to a pod only when it holds the entire machine, so a cluster can never share a host with another tenant.
  • Reserved together. Every node you select is reserved in one order. If one node cannot be rented, the whole order is rolled back: the pods already created for it are deleted in the same request. After the order is accepted, each node bills on its own. If a node fails to start, delete the cluster so the other nodes stop billing.

The price shown in the header is the whole order per hour, and it follows your row selection — deselect a machine and the price drops.

What you get when it starts​

Each node comes up as a normal pod with its own SSH access, plus three things an ordinary pod does not have.

A private overlay network. Every node gets an address on 10.42.0.0/24 — node 0 is 10.42.0.1, node 1 is 10.42.0.2, and so on — carried by a WireGuard mesh that exists only between your own pods. NCCL needs one routable address per rank to bootstrap its ring; on the default Docker bridge every container believes it is 172.17.0.2 and a rank ends up dialling itself.

The overlay is a control path, not a data path. Inside a pod, kernel WireGuard runs one tunnel on one CPU core, so wg0 carries about 1 Gb/s of TCP between two nodes. More streams do not help: 8 streams move the same total as 1. Use it for the rendezvous, SSH and small messages. NCCL collectives belong on the fabric: only RDMA verbs code such as NCCL can reach it, because the fabric NIC has no netdev in the pod. On one RoCE cluster, measured in September 2026, NCCL over RoCE ran 7.5 to 12 times faster than NCCL over sockets on wg0. A node to node copy over the overlay runs at about 1 Gb/s, so a 100 GB checkpoint takes about 15 minutes.

The RDMA devices. /dev/infiniband/ holds rdma_cm and one uverbs* entry per port — the same path for both wires, since RoCE speaks the same verbs API — so NCCL can use the fabric directly, with GPUDirect RDMA where the hardware supports it.

On RoCE the pod also sets NCCL_IB_HCA and NCCL_IB_GID_INDEX, because such a host answers verbs on every Ethernet card it has, including the one carrying the internet, and only one GID of the right card reaches the peer. A wrong pick does not fail, it halves your bandwidth. SSH sessions and nested containers get both, so env | grep NCCL_IB shows them wherever you run.

A shared login for the group. One key pair is minted for the rental and installed on every node, so ssh 10.42.0.2 from any member lands on that peer as root with nothing to set up and no host-key prompt. That is the piece mpirun and DeepSpeed's pdsh launcher were missing; neither tool ships in the image, so install the one you want before you launch with it.

Check all three from inside any node:

ip -4 addr show wg0          # 10.42.0.1/24 on the first node
wg show                      # one peer per other node, with a recent handshake
ls /dev/infiniband/          # rdma_cm  uverbs0 … uverbs7
ping -c3 10.42.0.2           # reach a peer over the overlay
ssh 10.42.0.2 hostname       # log in to a peer as root

If wg show lists no peer, or the ping fails, the fabric did not come up — see Troubleshooting.

Run a job across the nodes​

The launcher has to run on every node at once. With torchrun, each node gets the same command with its own --node_rank, and every node points at node 0's overlay address as the rendezvous host:

# on node 0 (the pod whose wg0 is 10.42.0.1)
torchrun --nnodes=2 --node_rank=0 --nproc_per_node=8 \
  --master_addr=10.42.0.1 --master_port=29500 train.py
# on node 1
torchrun --nnodes=2 --node_rank=1 --nproc_per_node=8 \
  --master_addr=10.42.0.1 --master_port=29500 train.py

Start them in either order but keep both running: torchrun prints nothing while it waits for the other ranks, so a single side on its own looks like it has hung.

--master_addr must be the overlay address, never the pod's public IP. Use a fresh --master_port for each run — a port left behind by a previous attempt makes the next one hang.

Confirm the traffic is on the fabric​

Set NCCL_DEBUG=INFO on the first run and look for the transport NCCL picked:

NCCL INFO NET/IB : Using [0]mlx5_0:1/IB … [7]mlx5_7:1/IB [RO]; OOB wg0:10.42.0.1<0>
NCCL INFO Channel 00/0 : 0[0] -> 1[0] [send] via NET/IB/4/GDRDMA

NET/IB is the fabric. A RoCE cluster prints the same transport with the card named differently:

NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE … ; OOB wg0:10.42.0.1<0>

If you see NET/Socket instead, NCCL fell back to TCP over the overlay and your job will run at a small fraction of the speed it should.

Bring your own image​

At the top level a cluster runs the Lium cluster image. Its entrypoint is what raises the overlay from the injected configuration and exports NCCL_SOCKET_IFNAME=wg0; on any other image the configuration would arrive and nothing would read it, so the rental would report success while NCCL quietly failed.

Your own image runs inside it. The pod ships a Docker daemon whose default runtime injects the RDMA devices into every container it starts, so a nested container gets the fabric with no extra flags:

docker run --rm --gpus all --network host \
  your-registry/your-image:tag  your-command

Do not omit --network host: it puts the container in the pod's network namespace, so it can see wg0 and reach the other nodes. The same default runtime copies the overlay's settings, including NCCL_SOCKET_IFNAME=wg0 and, on a RoCE cluster, the card and GID it picked, into every container that shares that namespace, so you do not pass them yourself. A variable you set with -e is left as you set it.

Stop a cluster​

Every node of a cluster bills on its own until it is removed. Deleting the nodes one at a time from Your Pods works, but a node you miss keeps billing. The API removes the whole group in one call: DELETE /clusters/<cluster_id> with your API key, where cluster_id is the value every member pod shows in the pod list (GET /pods). Members are torn down and billed exactly as a single pod delete does, one final settlement per node.

The answer lists one row per node with success and a message: "Pod deletion started" while the node still has to confirm, "Pod deletion is in progress" when the node was already being removed. success at the top is true only when every node accepted the delete. A node that refused stays in the list with its error, and the other nodes are still removed, so call again to retry the one that failed. An id nobody owns answers 404. A cluster of another account answers 403, or 404 when your key or session acts inside a workspace. A node still being created is sent its delete like the others; the node cancels the create. If the platform is ever set not to cancel in-flight creates, such a cluster answers 409 with the pending pod ids and nothing is removed until you try again.

The route is not behind the cluster feature flag: whoever holds a cluster can always stop it.

Current limits​

  • Whole hosts only, so a cluster cannot be assembled from partially free machines.
  • One GPU model at one density per cluster — see above for why.
  • Two nodes minimum. One machine is an ordinary pod with no fabric, and the order is refused.
  • The overlay needs UDP port 51820 between the machines. On a RoCE cluster the nodes reach each other over the fabric they already share, so nothing is open to the internet. On InfiniBand they use their public addresses. A provider who keeps that port closed does not appear in the Clusters tab.
  • A RoCE fabric is inferred from the provider's network. One provider, one IPv4 segment, one address per host. That says the cards are neighbours, not that the switch between them carries RDMA. InfiniBand has no such gap: its subnet manager answers for the fabric.
  • A RoCE fabric is measured before it is offered. While its nodes are free, Lium pushes RDMA traffic between every pair of them over the same card and GID your job will use. A pair the measurement does not cover is dropped from the listing, so an unverified RoCE cluster never reaches this tab.

Troubleshooting​

NCCL logs NET/Socket. NCCL did not find the fabric (a working cluster logs NET/IB). Check ls /dev/infiniband/ inside the container that runs the job — if it is empty in a nested container, the container was started with a runtime override that dropped the devices; drop the --runtime flag and let the default apply. On RoCE with the devices present, suspect the switch: without lossless queueing it carries no RDMA and NCCL has nowhere to go but TCP. Nothing inside your pod fixes that, so tell support which cluster it was.

The job just sits there. Every rank must be running. torchrun waits silently for the missing ones, which looks identical to a hang. Confirm the process is alive on every node, and that all of them use the same --master_addr, --master_port and --nnodes.

ping 10.42.0.2 gets no answer. The overlay is down. Run wg show on both nodes. For each peer it prints an endpoint address, and it says whether the two nodes ever shook hands. No handshake means UDP port 51820 does not get through to that endpoint address.

A node exits right after it starts. A cluster pod that starts with no active RDMA port stops itself before the workload runs. That is the machine, not your image: report the node and rent another fabric.

A cluster disappeared from the Clusters tab. It dropped below two free machines: one of them was rented, or stopped being entirely free. The tab only lists fabrics you can order right now.