Skip to main content

GPU Profiling (ncu)

GPU Profiling is an opt-in node capability: open the GPU performance counters on the host, and renters can run NVIDIA Nsight Compute (ncu) inside the pods they rent from you. The switch is a single host driver flag — there is no portal toggle. Once the platform detects the flag, the node gets a teal GPU Profiling badge in the marketplace and starts matching the renter-side GPU Profiling (ncu) filter.

Why enable it​

On a default driver configuration, NVIDIA restricts GPU performance counters to admin users, so ncu inside a rented pod fails with ERR_NVGPUCTRPERM. Machines with open counters are scarce on every GPU cloud, and renters do ask for them — kernel and inference-engine developers need ncu to tune CUDA code, and the demand is strongest on the latest silicon (B200 / B300), where kernels are still being optimized for the new architecture.

The upside is demand-side — your node becomes visible to a renter segment that ordinary nodes cannot serve at all. On flagship nodes it is also one of the three ways to keep the idle payout: an 8× H200, B200, or B300 node must offer GPU Profiling, GPU splitting, or run inside an attested confidential VM to earn the unrented incentive; with none of the three, the node stays active and rentable but forfeits that incentive while idle.

The trade-off: whole-host rentals only​

GPU performance counters are host-wide. With the flag on, any workload on the machine can read the counters — including a co-tenant on a split rental, who could watch a neighbor's GPU activity through them. To keep tenants isolated, Lium leases a profiling-enabled node only as a whole host:

  • GPU splitting is suspended while the flag is on — every rental takes all GPUs;
  • renters see the node in the GPU Profiling (ncu) filter only while it has no active rental.

Idle-time default jobs keep running while the node has no renter; the platform clears them before a renter's pod starts, as on any node.

Enable the flag if whole-node rentals fit your fleet. A node that mostly earns from 1×/2× split rentals loses that segment while the counters are open.

Enable the counters​

Only change the flag on an idle node

Opening the counters while rented pods are running exposes those renters to the side-channel above and trips a platform alert. Wait until the node has no active rentals (or drain it first).

The commands below are for the Ubuntu host image Lium nodes ship with; on another distribution use its own initramfs tool.

  1. Set the driver module option on the host:
echo 'options nvidia NVreg_RestrictProfilingToAdminUsers=0' | sudo tee /etc/modprobe.d/nvidia-profiling.conf
  1. Rebuild the initramfs and reboot:
sudo update-initramfs -u
sudo reboot
  1. Verify the loaded state after reboot:
grep RmProfilingAdminOnly /proc/driver/nvidia/params

Expected output:

RmProfilingAdminOnly: 0

The validator reads the loaded driver state on its next hardware scan of the node. Expect the GPU Profiling badge on lium.io within an hour, not instantly.

Detection is fail-closed: if the platform cannot read the state (no NVIDIA driver, unreadable /proc/driver/nvidia/params), the node is treated as not profiling-capable.

Disable the counters​

Remove the option and reboot — again, only while the node has no active rentals:

sudo rm /etc/modprobe.d/nvidia-profiling.conf
sudo update-initramfs -u
sudo reboot

After the next hardware scan the badge disappears and the node returns to normal rental rules, including GPU splitting if configured.