Skip to main content

Managing Nodes

This guide covers all aspects of node management, from adding new machines to configuring settings and handling customer requests.

Browserless / agent-driven node management

Every action in this page is also available from the lium provider CLI — same surface as the portal frontend, scriptable, with --json envelopes for AI agents and CI. For example: lium provider node add --gpu-type "NVIDIA H200 NVL" --gpu-count 8 --ip <IP> --port 8080 --price 1.85 --yes.

Adding New Nodes​

Prerequisites​

Before adding a node, ensure:

  1. Node Software Running: Your GPU machine has the node program running
  2. Network Access: Machine is accessible from the internet
  3. Provider connected: Your provider (self-hosted, or the Lium.io Central Provider Server) is running and connected to the portal

Registration Process​

  1. Access Add Node Modal:

    • Navigate to the Nodes page
    • Click the "Add Node" button
    • A popup modal will appear

  2. Fill Required Information:

    • GPU Type: Select from available options (RTX 4090, A100, etc.)
    • GPU Count: Enter the number of GPUs in the machine
    • Machine IP Address: Public IP address of your node
    • Port: Port number where the node program is running
    • GPU Price: Hourly price per GPU (in USD)

  3. Submit Registration:

    • Review all information for accuracy
    • Click "Add Node" to register

Synchronization​

The sync controls below apply when you run a self-hosted provider. If you've opted into the Lium.io Central Provider Server, Lium handles sync for you and these buttons are not needed.

Sync From the self-hosted provider​

Sync nodes from your self-hosted provider to the portal:

  • Purpose: Import node configurations from your self-hosted provider
  • When to Use: After setting up nodes directly on your server
  • Process: Click "Sync From Provider Server" button
  • Result: Nodes configured on your server appear in the portal
Note

When you first login on portal, the sync process will be done automatically for you.

Sync Into the self-hosted provider​

Sync node configurations from the portal to your self-hosted provider:

  • Purpose: Export portal configurations to your self-hosted provider
  • When to Use: After making changes in the portal that need to be reflected on your server if your server lost connection at that moment.
  • Process: Click "Sync Into Provider Server" button
  • Result: Portal configurations are applied to your self-hosted provider
Note

The syncing process is done automatically as long as your self-hosted provider is connected to the portal backend and you barely need to do manual sync at all.

Synchronization

Always ensure your self-hosted provider is running and accessible before attempting synchronization.

Post-Registration Steps​

After successful registration:

  1. Monitor Status: Check that the node appears as active
  2. Configure Settings: Set notice periods and other parameters

Price Management​

Updating Prices​

  1. Access Price Settings:

    • Navigate to your node details page
    • Click the "Update Price" button
    • Enter the new hourly rate per GPU (in USD) in the GPU Price field
    • Note: The Machine Price field is deprecated and will be ignored

  2. Price Considerations:

    • Minimum Viable Price: Set prices that cover your operational costs
    • Competitive Pricing: Research market rates to stay competitive
    • GPU Count Impact: Total rental cost = GPU Price × Number of GPUs rented
Price Updates

Price changes take effect immediately and apply to all new rental requests. Existing active rentals continue at their original rate until they end.

Pausing New Rentals​

Use Pause New Rentals when a node is currently rented and you want it to stop accepting future rentals after the current renter is done. This does not stop, delete, or interrupt the active pod.

When to use it​

  • You plan to take the machine offline after the current rental ends
  • You want to drain a node before maintenance or hardware changes
  • You want the current renter to finish normally, but do not want another renter to immediately take the node
Rented nodes stay active

While the pause is pending, the current rental remains active and the node is still checked by validators. The node is removed from new-rental availability immediately, including partial GPU rentals on GPU-splitting nodes.

Request a pause​

  1. Open the node details page in the Provider Portal.
  2. Click Pause New Rentals.
  3. Confirm the action.

After the request is saved:

  • The node no longer appears as available for new rentals.
  • Existing pods on the node continue to run.
  • The node shows a pending pause state until the last active pod on that node ends.

After the current rental ends​

When the final active pod on the node is deleted, the pause becomes active. At that point:

  • The node stays hidden from new-rental availability.
  • The node is not eligible for unrented availability rewards.
  • The node can be brought back with Resume New Rentals.

Cancel or resume​

  • Use Cancel New Rentals Pause while the pause is still pending and the node is still rented.
  • Use Resume New Rentals after the pause has fully applied and you want the node to accept rentals again.

Price limits​

Three limits apply to the price you can set; only the first two reject the price:

ConstraintEffect
Price below machine_min_price_rate × the GPU model's reference pricePortal rejects with HTTP 400 — the price is not saved. Portal-only: nothing else on the platform checks the floor
Price above machine_max_price_rate × the GPU model's reference pricePortal rejects with HTTP 400 — the price is not saved. Enforced beyond the portal: the platform refuses to save or refresh a node reporting a price above the cap, and the validator scores that node 0
Price above the GPU model's market p90 × soft_limit_price_rate (the soft limit)Nothing is rejected — the node stays rentable, but while idle it earns no unrented incentive. See Price and the unrented incentive

Both the reference prices and the floor/cap multipliers (0.5 and 2.5 today) change as the platform adds GPU models or rebalances rates, so read the limits from the API rather than from this page. The machines endpoint applies the multipliers for you — each GPU model carries its reference price in USD/hr, the allowed range as p_min and p_max, and the soft limit as soft_price_limit (null when the model has no market p90, so none applies):

curl https://lium.io/api/machines

GPU Splitting​

GPU splitting allows you to rent individual GPUs from a single node to multiple customers simultaneously, enabling more efficient resource utilization and flexible pricing.

Prerequisites​

GPU splitting requires Docker to run on overlay2 with an XFS backing filesystem (pquota, ftype=1). Complete the host-side setup from Docker Storage Setup before continuing — the Edit button on the GPU Splitting panel stays disabled until preflight reports the host as ready.

Setting Minimum GPU Count for Rental​

After configuring the prerequisites, you must set a minimum GPU count to enable GPU splitting on your node.

  1. Access GPU Splitting Settings:

    • Navigate to your node details page in the portal
    • Locate the "GPU Splitting" section in the details panel
    • If your node meets all prerequisites, you will see an "Edit" button

  2. Configure Minimum GPU Count:

    • Click the "Edit" button
    • Set the minimum GPU count (must be greater than 1)
    • Save your changes
Minimum GPU Count

GPU splitting requires a minimum GPU count to be configured. If the minimum GPU count is not set, the node will not support GPU splitting, and customers will only be able to rent all GPUs on the node. Once configured, customers can rent any number of GPUs from the minimum up to the total available GPUs on the node.

Disabling GPU Splitting: You can remove the minimum GPU count to disable GPU splitting. However, you can only remove it when there are no pods currently renting a partial number of GPUs.

Cluster Fabrics (InfiniBand and RoCE)​

Nodes with an RDMA card can be wired into one fabric, which is what a distributed job across several machines needs. Both wires are sold: InfiniBand, paired by the subnet manager both nodes report, and RoCE (RDMA over Ethernet), paired when two of your nodes answer on one IPv4 segment. The portal shows where each of your nodes stands, and customers rent a ready fabric as one order from the Clusters tab — see Multi-node clusters for what they get when they do.

A RoCE pair also needs the switch between the two nodes to carry RDMA: lossless queueing, PFC/ECN. Lium measures that link between your two free nodes before offering the pair, so a switch that drops RDMA means the pair is never sold as a cluster at all. Configure it before you offer the pair.

Cluster badges​

The Cluster status switch above the Nodes list shows or hides these badges; it covers both wires at once and the choice is remembered in your browser. With it on, the Nodes list and the node details page show one of:

  • CLUSTER READY +N (RoCE: ROCE CLUSTER READY +N) — this node shares a fabric with N of your other nodes. The details page links to them.
  • IB UNPAIRED / ROCE UNPAIRED — the link works, but the node has nothing to join: no other node of yours on the same fabric, or the fabric itself is not joinable. The tooltip names the reason and what fixes it — including a RoCE card that qualifies but holds no usable IPv4 address yet.
  • IB LINK DOWN / ROCE LINK DOWN — the card is installed but no port is active. The link is down or not cabled.

Nodes without an RDMA card show no badge, and nodes are only ever paired with your own. If you run several miner hotkeys, nodes registered under different hotkeys are treated as separate fabrics even when they sit behind the same switch. A node holding both a live InfiniBand port and a live RoCE card is sold on InfiniBand, and its badge reads the InfiniBand side.

A RoCE pair must also pass Lium's link measurement: the validator runs an RDMA transfer between your free nodes and records the result. The tooltip tells you where the pair stands — measured and ready, waiting for a measurement, or failed. A failed measurement points at the switch: enable lossless Ethernet (PFC/ECN) and the next cycle re-measures it.

On a RoCE node's details page you also get the speed measured to each of its cluster partners, slowest first, with the slowest one repeated at the top: a distributed job runs no faster than the weakest link, so that is the one worth fixing. The port rate shown beside it is what the card reports it can do — a measurement far below it means the network between the two nodes, not the card.

Making nodes cluster-ready​

  1. Bring the port up — ibstat must report an ACTIVE port. A DOWN or Polling port usually means the cable is missing or the card is not connected to the switch.
  2. Put the nodes on one fabric — connect them to the same switch, served by one subnet manager. A host running the subnet manager on its own port is a one-host subnet that nothing can join, and a host whose ports report different subnet managers is ambiguous and is not paired either. RoCE has no subnet manager: give the two cards one address each in the same /24, and put the switch in lossless mode. One address per host — when two of your nodes claim the same address, they sit behind different networks that only look alike, and the badge says so. A host with a live InfiniBand port is sold on InfiniBand and never on RoCE, even when it also carries a RoCE card, so a mixed host shows no RoCE pairing — its badge reads the InfiniBand side.
  3. Keep them in one partition — nodes in different partitions cannot talk even on one subnet, so the portal does not pair them. Read a port's partition key with cat /sys/class/infiniband/<device>/ports/<port>/pkeys/0; 0xffff is the default full-membership one, and a value below 0x8000 is a limited member. Two limited members cannot reach each other, so at least one node of a pair has to be a full member. RoCE has no partitions; skip this step there.
  4. Match the hardware — a cluster is rented as one GPU model at one density, because a single image runs against one architecture and an uneven number of GPUs per node paces the job by the smallest node. Two different cards on one fabric, or the same card at 8 GPUs on one node and 4 on another, are two separate groups.
  5. Keep the node whole — a node opened for partial rentals cannot join a cluster: the RDMA devices only reach a pod that holds the entire host. Clear the minimum GPU count for rental on the node to make it joinable.
  6. Open UDP 51820 — a cluster node carries a private overlay that the ranks use to find each other before the traffic moves onto the fabric. It listens on UDP port 51820 and needs to be reachable from your other nodes' public addresses. A blocked port fails late: the node looks cluster-ready here and then cannot join the job.
  7. Wait for the next validation — the badge follows the validator's node scrape, so it updates within roughly 15 minutes.
Why bother

Very little supply on the network can serve multi-node jobs at all, and a cluster is rented as one all-or-nothing order: every member is taken at once, as a whole host. Wiring your nodes together is the slow part, and it is done once.

Notice Period Management​

Understanding Notice Periods​

Notice periods allow you to schedule maintenance for rented machines while keeping customers informed:

  • Requirement
    • Notice period should be scheduled 24 hours ahead.
    • Maximum period is 60 mins
  • Purpose:
    • Give customers advance warning of planned downtime
    • Give providers a safe maintenance window for the machine

Setting Notice Periods​

  • Go to node management page
  • Find the "Notice Period" section
  • Select the start time. (minimum 24 hours later from now)
  • Enter desired notice period (maximum 60 mins)
Note

Once the notice period is scheduled, it will send an email to the customer who rented the machine and let them know.

Notify Machine Request from Customer​

Respond to customer requests efficiently:

  1. Access Notify Feature:

    • Go to your node details page
    • Click the "Notify" button in the top-right corner
    • A modal window displays available machine requests

  2. Select and Notify:

    • Review available machine requests
    • Select appropriate requests based on your node's capabilities
    • Click "Notify" to send notification to the renter

Force-closing a pod (Mark broken)​

Why this exists & when to use it​

Sometimes a pod's container disappears or dies on your host machine, but Lium still shows the pod as rented. When that happens the node gets stuck: the rental isn't actually running, so the node keeps scoring zero incentive, and you have no way to clear the dangling pod on your own — leaving the machine unusable even after a penalty has been applied.

Force-closing is the recovery path for exactly this situation. By marking the dead pod broken you accept the provider penalty and refund the renter what they paid; in return, the pod is cleared and your node returns to the rentable pool so it can be rented and earn its score again.

Use it when:

  • A rented pod's container is gone or dead on the host but still shows as rented/running in Lium, and
  • You have no other way to recover the node.

It's a recovery tool, not a routine action — it costs you a penalty and disrupts the renter, so only reach for it when the node is genuinely stuck.

How to do it​

  1. Open the node's details page in the Provider Portal.
  2. In the pod status table, find the pod and click Force close / Mark broken.
  3. Read the confirmation, then click Mark as broken.

What happens​

  • A provider penalty is applied: the rental earnings for that pod are declined from you and credited back to the renter for the disruption.
  • The pod's GPUs are released back to the rentable pool, so the node can be rented again and recover its score.
  • For the renter, the pod turns to BROKEN — they're notified they were credited and the pod can only be deleted.
The container is NOT torn down for you

Marking a pod broken only updates billing and state. The platform does not stop the container — you must kill it yourself on the host machine. Otherwise it keeps running and consuming resources.

Tier affects the penalty

On Secure nodes a force-close applies the penalty above. On Spot nodes there is no penalty (renters already accept that Spot pods may be interrupted). See Node Tier: Secure vs Spot.

Node Deletion​

When to Delete a Node​

Consider deleting a node when:

  • Hardware Issues: Machine is no longer functional
  • Upgrade Plans: Replacing with better hardware
  • Cost Optimization: Reducing operational costs
  • Relocation: Moving to a different location

Deletion Process​

  1. Delete Node:
    • Delete from the portal

Troubleshooting​

Common Issues​

Node Not Appearing​

  • Check Registration: Verify all required fields were filled correctly
  • Network Connectivity: Ensure machine is accessible from the internet
  • Provider connected: Confirm your provider is running and connected (self-hosted, or Lium.io Central Provider Server toggle on)
  • Validation: Wait for validation process (minimum 15 minutes) to complete