Skip to main content

Multi-node clusters (InfiniBand)

Rent several machines that sit on the same InfiniBand fabric and run one distributed job across all of them. A single pod caps you at the GPUs in one box; a cluster lets a training run span many, with the gradients travelling over InfiniBand instead of the public internet.

When you need one​

A cluster only helps when the job itself spans machines — data-parallel training whose model or batch no longer fits one host, tensor or pipeline parallelism, or a long run you want to finish sooner. Anything that fits inside one machine is cheaper and simpler as an ordinary pod: rent a single 8-GPU host from Browse Pods and skip this page.

Find a cluster​

  1. Open Browse Pods on lium.io and switch to the Clusters tab.
  2. Narrow with the GPU type and GPU count filters if you have a specific card in mind.
  3. Each card is one fabric: the GPU model and per-node GPU count in the header, the price and the rent button on the right, and one row per machine with its CPU, memory, VRAM, disk, network and location.

A card only appears when two or more entirely free machines share one fabric. If a fabric's machines are rented out, or only one is free, it is not a cluster you can order right now, so it is not shown.

What a cluster is, exactly​

Every cluster Lium sells is homogeneous and whole:

  • One GPU model, one GPU count per node. A fabric carrying both H100 and H200 machines is offered as two separate cards. Mixed hardware would pace every collective by the slowest card, and an uneven number of ranks per node breaks tensor and pipeline parallel sharding.
  • Whole machines only. You get every GPU on each host. The RDMA devices are handed to a pod only when it holds the entire machine, so a cluster can never share a host with another tenant.
  • All or nothing. Every node you select deploys together, or none does. If one node fails to start, the whole order is rolled back and you are not charged for the rest.

The price shown in the header is the whole order per hour, and it follows your row selection — deselect a machine and the price drops.

What you get when it starts​

Each node comes up as a normal pod with its own SSH access, plus two things an ordinary pod does not have.

A private overlay network. Every node gets an address on 10.42.0.0/24 — node 0 is 10.42.0.1, node 1 is 10.42.0.2, and so on — carried by a WireGuard mesh that exists only between your own pods. NCCL needs one routable address per rank to bootstrap its ring; on the default Docker bridge every container believes it is 172.17.0.2 and a rank ends up dialling itself.

The InfiniBand devices. /dev/infiniband/ holds rdma_cm and one uverbs* entry per port, so NCCL can use the fabric directly, with GPUDirect RDMA where the hardware supports it.

Check both from inside any node:

ip -4 addr show wg0          # 10.42.0.1/24 on the first node
wg show                      # one peer per other node, with a recent handshake
ls /dev/infiniband/          # rdma_cm  uverbs0 … uverbs7
ping -c3 10.42.0.2           # reach a peer over the overlay

If wg show lists no peer, or the ping fails, the fabric did not come up — see Troubleshooting.

Run a job across the nodes​

The launcher has to run on every node at once. With torchrun, each node gets the same command with its own --node_rank, and every node points at node 0's overlay address as the rendezvous host:

# on node 0 (the pod whose wg0 is 10.42.0.1)
torchrun --nnodes=2 --node_rank=0 --nproc_per_node=8 \
  --master_addr=10.42.0.1 --master_port=29500 train.py
# on node 1
torchrun --nnodes=2 --node_rank=1 --nproc_per_node=8 \
  --master_addr=10.42.0.1 --master_port=29500 train.py

Start them in either order but keep both running: torchrun prints nothing while it waits for the other ranks, so a single side on its own looks like it has hung.

--master_addr must be the overlay address, never the pod's public IP. Use a fresh --master_port for each run — a port left behind by a previous attempt makes the next one hang.

Confirm the traffic is on InfiniBand​

Set NCCL_DEBUG=INFO on the first run and look for the transport NCCL picked:

NCCL INFO NET/IB : Using [0]mlx5_0:1/IB … [7]mlx5_7:1/IB [RO]; OOB wg0:10.42.0.1<0>
NCCL INFO Channel 00/0 : 0[0] -> 1[0] [send] via NET/IB/4/GDRDMA

NET/IB is the fabric. If you see NET/Socket instead, NCCL fell back to TCP over the overlay and your job will run at a small fraction of the speed it should.

Bring your own image​

At the top level a cluster runs the Lium cluster image. Its entrypoint is what raises the overlay from the injected configuration and exports NCCL_SOCKET_IFNAME=wg0; on any other image the configuration would arrive and nothing would read it, so the rental would report success while NCCL quietly failed.

Your own image runs inside it. The pod ships a Docker daemon whose default runtime injects the RDMA devices into every container it starts, so a nested container gets the fabric with no extra flags:

docker run --rm --gpus all --network host \
  -e NCCL_SOCKET_IFNAME=wg0 \
  your-registry/your-image:tag  your-command

Two flags matter. --network host puts the container in the pod's network namespace so it can see wg0 and reach the other nodes. -e NCCL_SOCKET_IFNAME=wg0 is needed because the variable is not inherited automatically — without it NCCL guesses which interface to bootstrap on and may pick the wrong one.

Current limits​

  • Whole hosts only, so a cluster cannot be assembled from partially free machines.
  • One GPU model at one density per cluster — see above for why.
  • Two nodes minimum. One machine is an ordinary pod with no fabric, and the order is refused.
  • Nodes cannot SSH each other yet. Your key gives you access from outside, but there is no key inside the pods, so mpirun, pdsh and DeepSpeed's default launcher — all of which start remote processes over SSH — do not work between nodes. Launch with torchrun per node as shown above, or install your own key across the pods.
  • UDP 51820 must be reachable on each machine for the overlay. Providers who have not opened it will not appear as cluster-capable.

Troubleshooting​

NET/Socket instead of NET/IB. NCCL did not find the fabric. Check ls /dev/infiniband/ inside the container that runs the job — if it is empty in a nested container, the container was started with a runtime override that dropped the devices; drop the --runtime flag and let the default apply.

The job just sits there. Every rank must be running. torchrun waits silently for the missing ones, which looks identical to a hang. Confirm the process is alive on every node, and that all of them use the same --master_addr, --master_port and --nnodes.

ping 10.42.0.2 fails. The overlay is down. wg show on both nodes tells you whether the peer handshake ever happened; a peer with no handshake usually means UDP 51820 is blocked between the two machines.

A cluster disappeared from the Clusters tab. Someone rented one of its machines, or a node stopped being entirely free. The tab only lists fabrics you can order right now.