Skip to main content

Multi-node clusters

A cluster is several whole machines on the same RDMA fabric, for one distributed job. If your job fits on one machine, rent a single pod instead.

Rent a cluster​

Find one

On lium.io, open Browse Pods and switch to the Clusters tab. Filter by GPU type and GPU count.

Read the card

Each card is one fabric: GPU model, GPUs per node, location and price. A badge shows InfiniBand or RoCE. Every RoCE cluster on the tab has been tested for RDMA before it is listed.

Rent it

Pick the machines (at least two) and click rent. The price shown is for the whole order per hour. All nodes have the same GPU model and count.

The cluster runs the Multi-node cluster template. If you pick another Lium template, the cluster template is used instead. Every node must report a GPU driver that supports the cluster template's CUDA version, or the rental is refused.

From the terminal: lium clusters up <fabric> --nodes N --name <name> (CLI reference).

What each node gets​

Each node is a normal pod with SSH, plus:

  • A private network: node 0 is 10.42.0.1, node 1 is 10.42.0.2, and so on. Use it for setup and SSH, not for heavy data (about 1 Gb/s).
  • The RDMA devices in /dev/infiniband/, so NCCL uses the fabric.
  • Shared SSH: ssh 10.42.0.2 from any node logs in to that node.

Check it works

ip -4 addr show wg0          # 10.42.0.1/24 on the first node
ls /dev/infiniband/          # rdma_cm  uverbs0 … uverbs7
ssh 10.42.0.2 hostname       # log in to a peer as root

Run a job​

Run the same command on every node with its own --node_rank. Point --master_addr at node 0's private address:

# on node 0 (the pod whose wg0 is 10.42.0.1)
torchrun --nnodes=2 --node_rank=0 --nproc_per_node=8 \
  --master_addr=10.42.0.1 --master_port=29500 train.py
# on node 1
torchrun --nnodes=2 --node_rank=1 --nproc_per_node=8 \
  --master_addr=10.42.0.1 --master_port=29500 train.py

In Your Pods, click Launch info on the cluster block to copy the hostfile and one torchrun command per node.

torchrun prints nothing until every node has started.

Confirm it uses the fabric

Set NCCL_DEBUG=INFO. The log should show NET/IB. If it shows NET/Socket, NCCL is not using the fabric; tell support which cluster it is.

Use your own image​

You can rent a cluster with your own template. Build its image FROM daturaai/lium-cluster: that image starts the private network and the shared SSH. On another image, the nodes cannot find each other.

You can also run your image inside the cluster pod with Docker. Keep --network host:

docker run --rm --gpus all --network host \
  your-registry/your-image:tag  your-command

Replace a failed node​

Replace it

In Your Pods, a node that is down shows Replace node N on the cluster block. Click it. The API does the same with POST /clusters/<cluster_id>/members/<N>/replace.

What happens

Lium rents a free node from the same provider and fabric. It keeps the failed node's number, private address, shared SSH and scheduled stop time, so the other nodes need no change. The failed node is deleted once the new one is running.

If no such node is free, the request is refused; try again later or rent a new cluster. The new node starts empty: copy your data back to it.

Stop a cluster​

If a node fails to start within 20 minutes, Lium deletes the whole cluster, and a node that never started is not charged.

Each node bills on its own until it is deleted. In Your Pods, click Delete cluster on the cluster block and confirm to delete every node at once. The CLI does the same with lium clusters rm <cluster>, and the API with DELETE /clusters/<cluster_id> and your API key. cluster_id is shown on each member pod in GET /pods.

Last updated on