--- sidebar_position: 4.5 --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.lium.io/llms.txt > Use this file to discover all available pages before exploring further. # Multi-node clusters Rent several machines that sit on the same RDMA fabric and run one distributed job across all of them. A single pod caps you at the GPUs in one box; a cluster lets a training run span many, with the gradients travelling over the fabric instead of the public internet. ## When you need one A cluster only helps when the job itself spans machines — data-parallel training whose model or batch no longer fits one host, tensor or pipeline parallelism, or a long run you want to finish sooner. Anything that fits inside one machine is cheaper and simpler as an ordinary pod: rent a single 8-GPU host from **Browse Pods** and skip this page. ## Find a cluster 1. Open **Browse Pods** on [lium.io](https://lium.io) and switch to the **Clusters** tab. 2. Narrow with the **GPU type** and **GPU count** filters if you have a specific card in mind. 3. Each card is one fabric: the GPU model and per-node GPU count in the header, a badge saying which wire links it, the location, the price and the rent button on the right, and one row per machine with its CPU, memory, VRAM, disk and network. A card appears when **two or more machines** on one fabric are free to rent right now. The rows list those machines — the fabric itself can be larger, because its other machines are already rented. A fabric with only one free machine shows no card; that machine is still on sale in the **List** tab as an ordinary pod. The location is on the card, not on the rows: one fabric is one switch, so its machines are in one rack. The location comes from the address the provider registered, so read it as the region, not as the street. ## InfiniBand or RoCE The badge on each cluster card, and on a node in the **List** tab, says which wire you are buying. Everything else on this page is the same for both. - **InfiniBand** — its own switch and subnet manager. Both nodes report the same subnet manager, so the fabric between them is a measured fact. - **RoCE** — RDMA over Ethernet. Same rental, same overlay, same speed on paper. The grouping is weaker: one provider plus one IPv4 segment, which is usually a shared switch. A switch without lossless queueing (PFC/ECN) carries no RDMA and NCCL drops to TCP. Lium therefore measures the link before it offers the pair, so every RoCE cluster on the tab is one we pushed RDMA traffic across. A pair that fails the measurement is not offered at all. ## What a cluster is, exactly Every cluster Lium sells is homogeneous and whole: - **One GPU model, one GPU count per node.** A fabric carrying both H100 and H200 machines is offered as two separate cards. Mixed hardware would pace every collective by the slowest card, and an uneven number of ranks per node breaks tensor and pipeline parallel sharding. - **Whole machines only.** You get every GPU on each host. The RDMA devices are handed to a pod only when it holds the entire machine, so a cluster can never share a host with another tenant. - **All or nothing.** Every node you select deploys together, or none does. If one node fails to start, the whole order is rolled back and you are not charged for the rest. The price shown in the header is the **whole order** per hour, and it follows your row selection — deselect a machine and the price drops. ## What you get when it starts Each node comes up as a normal pod with its own SSH access, plus three things an ordinary pod does not have. **A private overlay network.** Every node gets an address on `10.42.0.0/24` — node 0 is `10.42.0.1`, node 1 is `10.42.0.2`, and so on — carried by a WireGuard mesh that exists only between your own pods. NCCL needs one routable address per rank to bootstrap its ring; on the default Docker bridge every container believes it is `172.17.0.2` and a rank ends up dialling itself. **The RDMA devices.** `/dev/infiniband/` holds `rdma_cm` and one `uverbs*` entry per port — the same path for both wires, since RoCE speaks the same verbs API — so NCCL can use the fabric directly, with GPUDirect RDMA where the hardware supports it. On RoCE the pod also sets `NCCL_IB_HCA` and `NCCL_IB_GID_INDEX`, because such a host answers verbs on every Ethernet card it has, including the one carrying the internet, and only one GID of the right card reaches the peer. A wrong pick does not fail, it halves your bandwidth. SSH sessions and nested containers get both, so `env | grep NCCL_IB` shows them wherever you run. **A shared login for the group.** One key pair is minted for the rental and installed on every node, so `ssh 10.42.0.2` from any member lands on that peer as root with nothing to set up and no host-key prompt. That is the piece `mpirun` and DeepSpeed's pdsh launcher were missing; neither tool ships in the image, so install the one you want before you launch with it. Check all three from inside any node: ```bash ip -4 addr show wg0 # 10.42.0.1/24 on the first node wg show # one peer per other node, with a recent handshake ls /dev/infiniband/ # rdma_cm uverbs0 … uverbs7 ping -c3 10.42.0.2 # reach a peer over the overlay ssh 10.42.0.2 hostname # log in to a peer as root ``` If `wg show` lists no peer, or the ping fails, the fabric did not come up — see [Troubleshooting](#troubleshooting). ## Run a job across the nodes The launcher has to run on every node at once. With `torchrun`, each node gets the same command with its own `--node_rank`, and every node points at node 0's overlay address as the rendezvous host: ```bash # on node 0 (the pod whose wg0 is 10.42.0.1) torchrun --nnodes=2 --node_rank=0 --nproc_per_node=8 \ --master_addr=10.42.0.1 --master_port=29500 train.py ``` ```bash # on node 1 torchrun --nnodes=2 --node_rank=1 --nproc_per_node=8 \ --master_addr=10.42.0.1 --master_port=29500 train.py ``` Start them in either order but keep both running: `torchrun` prints nothing while it waits for the other ranks, so a single side on its own looks like it has hung. `--master_addr` must be the **overlay** address, never the pod's public IP. Use a fresh `--master_port` for each run — a port left behind by a previous attempt makes the next one hang. ### Confirm the traffic is on the fabric Set `NCCL_DEBUG=INFO` on the first run and look for the transport NCCL picked: ``` NCCL INFO NET/IB : Using [0]mlx5_0:1/IB … [7]mlx5_7:1/IB [RO]; OOB wg0:10.42.0.1<0> NCCL INFO Channel 00/0 : 0[0] -> 1[0] [send] via NET/IB/4/GDRDMA ``` `NET/IB` is the fabric. A RoCE cluster prints the same transport with the card named differently: ``` NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE … ; OOB wg0:10.42.0.1<0> ``` If you see `NET/Socket` instead, NCCL fell back to TCP over the overlay and your job will run at a small fraction of the speed it should. ## Bring your own image At the top level a cluster runs the Lium cluster image. Its entrypoint is what raises the overlay from the injected configuration and exports `NCCL_SOCKET_IFNAME=wg0`; on any other image the configuration would arrive and nothing would read it, so the rental would report success while NCCL quietly failed. Your own image runs **inside** it. The pod ships a Docker daemon whose default runtime injects the RDMA devices into every container it starts, so a nested container gets the fabric with no extra flags: ```bash docker run --rm --gpus all --network host \ your-registry/your-image:tag your-command ``` Do not omit `--network host`: it puts the container in the pod's network namespace, so it can see `wg0` and reach the other nodes. The same default runtime copies the overlay's settings, including `NCCL_SOCKET_IFNAME=wg0` and, on a RoCE cluster, the card and GID it picked, into every container that shares that namespace, so you do not pass them yourself. A variable you set with `-e` is left as you set it. ## Current limits - **Whole hosts only**, so a cluster cannot be assembled from partially free machines. - **One GPU model at one density per cluster** — see above for why. - **Two nodes minimum.** One machine is an ordinary pod with no fabric, and the order is refused. - **The overlay needs UDP port 51820 between the machines.** On a RoCE cluster the nodes reach each other over the fabric they already share, so nothing is open to the internet. On InfiniBand they use their public addresses. A provider who keeps that port closed does not appear in the Clusters tab. - **A RoCE fabric is inferred from the provider's network.** One provider, one IPv4 segment, one address per host. That says the cards are neighbours, not that the switch between them carries RDMA. InfiniBand has no such gap: its subnet manager answers for the fabric. - **A RoCE fabric is measured before it is offered.** While its nodes are free, Lium pushes RDMA traffic between every pair of them over the same card and GID your job will use. A pair the measurement does not cover is dropped from the listing, so an unverified RoCE cluster never reaches this tab. ## Troubleshooting **`NET/Socket` instead of `NET/IB`.** NCCL did not find the fabric. Check `ls /dev/infiniband/` inside the container that runs the job — if it is empty in a nested container, the container was started with a runtime override that dropped the devices; drop the `--runtime` flag and let the default apply. On RoCE with the devices present, suspect the switch: without lossless queueing it carries no RDMA and NCCL has nowhere to go but TCP. Nothing inside your pod fixes that, so tell support which cluster it was. **The job just sits there.** Every rank must be running. `torchrun` waits silently for the missing ones, which looks identical to a hang. Confirm the process is alive on every node, and that all of them use the same `--master_addr`, `--master_port` and `--nnodes`. **`ping 10.42.0.2` gets no answer.** The overlay is down. Run `wg show` on both nodes. For each peer it prints an endpoint address, and it says whether the two nodes ever shook hands. No handshake means UDP port 51820 does not get through to that endpoint address. **A node exits right after it starts.** A cluster pod with no active RDMA port refuses to run rather than train over TCP at cluster price. That is the machine, not your image: report the node and rent another fabric. **A cluster disappeared from the Clusters tab.** It dropped below two free machines: one of them was rented, or stopped being entirely free. The tab only lists fabrics you can order right now.