--- sidebar_position: 4.5 --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.lium.io/llms.txt > Use this file to discover all available pages before exploring further. # Multi-node clusters (InfiniBand) Rent several machines that sit on the same InfiniBand fabric and run one distributed job across all of them. A single pod caps you at the GPUs in one box; a cluster lets a training run span many, with the gradients travelling over InfiniBand instead of the public internet. ## When you need one A cluster only helps when the job itself spans machines — data-parallel training whose model or batch no longer fits one host, tensor or pipeline parallelism, or a long run you want to finish sooner. Anything that fits inside one machine is cheaper and simpler as an ordinary pod: rent a single 8-GPU host from **Browse Pods** and skip this page. ## Find a cluster 1. Open **Browse Pods** on [lium.io](https://lium.io) and switch to the **Clusters** tab. 2. Narrow with the **GPU type** and **GPU count** filters if you have a specific card in mind. 3. Each card is one fabric: the GPU model and per-node GPU count in the header, the price and the rent button on the right, and one row per machine with its CPU, memory, VRAM, disk, network and location. A card only appears when **two or more entirely free machines** share one fabric. If a fabric's machines are rented out, or only one is free, it is not a cluster you can order right now, so it is not shown. ## What a cluster is, exactly Every cluster Lium sells is homogeneous and whole: - **One GPU model, one GPU count per node.** A fabric carrying both H100 and H200 machines is offered as two separate cards. Mixed hardware would pace every collective by the slowest card, and an uneven number of ranks per node breaks tensor and pipeline parallel sharding. - **Whole machines only.** You get every GPU on each host. The RDMA devices are handed to a pod only when it holds the entire machine, so a cluster can never share a host with another tenant. - **All or nothing.** Every node you select deploys together, or none does. If one node fails to start, the whole order is rolled back and you are not charged for the rest. The price shown in the header is the **whole order** per hour, and it follows your row selection — deselect a machine and the price drops. ## What you get when it starts Each node comes up as a normal pod with its own SSH access, plus two things an ordinary pod does not have. **A private overlay network.** Every node gets an address on `10.42.0.0/24` — node 0 is `10.42.0.1`, node 1 is `10.42.0.2`, and so on — carried by a WireGuard mesh that exists only between your own pods. NCCL needs one routable address per rank to bootstrap its ring; on the default Docker bridge every container believes it is `172.17.0.2` and a rank ends up dialling itself. **The InfiniBand devices.** `/dev/infiniband/` holds `rdma_cm` and one `uverbs*` entry per port, so NCCL can use the fabric directly, with GPUDirect RDMA where the hardware supports it. Check both from inside any node: ```bash ip -4 addr show wg0 # 10.42.0.1/24 on the first node wg show # one peer per other node, with a recent handshake ls /dev/infiniband/ # rdma_cm uverbs0 … uverbs7 ping -c3 10.42.0.2 # reach a peer over the overlay ``` If `wg show` lists no peer, or the ping fails, the fabric did not come up — see [Troubleshooting](#troubleshooting). ## Run a job across the nodes The launcher has to run on every node at once. With `torchrun`, each node gets the same command with its own `--node_rank`, and every node points at node 0's overlay address as the rendezvous host: ```bash # on node 0 (the pod whose wg0 is 10.42.0.1) torchrun --nnodes=2 --node_rank=0 --nproc_per_node=8 \ --master_addr=10.42.0.1 --master_port=29500 train.py ``` ```bash # on node 1 torchrun --nnodes=2 --node_rank=1 --nproc_per_node=8 \ --master_addr=10.42.0.1 --master_port=29500 train.py ``` Start them in either order but keep both running: `torchrun` prints nothing while it waits for the other ranks, so a single side on its own looks like it has hung. `--master_addr` must be the **overlay** address, never the pod's public IP. Use a fresh `--master_port` for each run — a port left behind by a previous attempt makes the next one hang. ### Confirm the traffic is on InfiniBand Set `NCCL_DEBUG=INFO` on the first run and look for the transport NCCL picked: ``` NCCL INFO NET/IB : Using [0]mlx5_0:1/IB … [7]mlx5_7:1/IB [RO]; OOB wg0:10.42.0.1<0> NCCL INFO Channel 00/0 : 0[0] -> 1[0] [send] via NET/IB/4/GDRDMA ``` `NET/IB` is the fabric. If you see `NET/Socket` instead, NCCL fell back to TCP over the overlay and your job will run at a small fraction of the speed it should. ## Bring your own image At the top level a cluster runs the Lium cluster image. Its entrypoint is what raises the overlay from the injected configuration and exports `NCCL_SOCKET_IFNAME=wg0`; on any other image the configuration would arrive and nothing would read it, so the rental would report success while NCCL quietly failed. Your own image runs **inside** it. The pod ships a Docker daemon whose default runtime injects the RDMA devices into every container it starts, so a nested container gets the fabric with no extra flags: ```bash docker run --rm --gpus all --network host \ -e NCCL_SOCKET_IFNAME=wg0 \ your-registry/your-image:tag your-command ``` Two flags matter. `--network host` puts the container in the pod's network namespace so it can see `wg0` and reach the other nodes. `-e NCCL_SOCKET_IFNAME=wg0` is needed because the variable is not inherited automatically — without it NCCL guesses which interface to bootstrap on and may pick the wrong one. ## Current limits - **Whole hosts only**, so a cluster cannot be assembled from partially free machines. - **One GPU model at one density per cluster** — see above for why. - **Two nodes minimum.** One machine is an ordinary pod with no fabric, and the order is refused. - **Nodes cannot SSH each other yet.** Your key gives you access from outside, but there is no key inside the pods, so `mpirun`, `pdsh` and DeepSpeed's default launcher — all of which start remote processes over SSH — do not work between nodes. Launch with `torchrun` per node as shown above, or install your own key across the pods. - **UDP 51820 must be reachable** on each machine for the overlay. Providers who have not opened it will not appear as cluster-capable. ## Troubleshooting **`NET/Socket` instead of `NET/IB`.** NCCL did not find the fabric. Check `ls /dev/infiniband/` inside the container that runs the job — if it is empty in a nested container, the container was started with a runtime override that dropped the devices; drop the `--runtime` flag and let the default apply. **The job just sits there.** Every rank must be running. `torchrun` waits silently for the missing ones, which looks identical to a hang. Confirm the process is alive on every node, and that all of them use the same `--master_addr`, `--master_port` and `--nnodes`. **`ping 10.42.0.2` fails.** The overlay is down. `wg show` on both nodes tells you whether the peer handshake ever happened; a peer with no handshake usually means UDP 51820 is blocked between the two machines. **A cluster disappeared from the Clusters tab.** Someone rented one of its machines, or a node stopped being entirely free. The tab only lists fabrics you can order right now.