---
sidebar_position: 7
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lium.io/llms.txt
> Use this file to discover all available pages before exploring further.

# GPU Profiling (ncu)

GPU Profiling is an opt-in node capability: open the GPU performance counters on the host, and renters can run [NVIDIA Nsight Compute](https://developer.nvidia.com/nsight-compute) (`ncu`) inside the pods they rent from you. The switch is a single host driver flag — there is no portal toggle. Once the platform detects the flag, the node gets a teal **GPU Profiling** badge in the marketplace and starts matching the renter-side **GPU Profiling (ncu)** filter.

## Why enable it

On a default driver configuration, NVIDIA restricts GPU performance counters to admin users, so `ncu` inside a rented pod fails with `ERR_NVGPUCTRPERM`. Machines with open counters are scarce on every GPU cloud, and renters do ask for them — kernel and inference-engine developers need `ncu` to tune CUDA code, and the demand is strongest on the latest silicon (B200 / B300), where kernels are still being optimized for the new architecture.

The upside is demand-side — your node becomes visible to a renter segment that ordinary nodes cannot serve at all. On flagship nodes it is also one of the two ways to keep the idle payout: an 8× H200, B200, or B300 node must offer GPU Profiling or [GPU splitting](./gpu-splitting.md) to earn the [unrented incentive](../rewards/emission.mdx#unrented-pool); with neither, the node stays active and rentable but forfeits that incentive while idle.

## The trade-off: whole-host rentals only

GPU performance counters are **host-wide**. With the flag on, any workload on the machine can read the counters — including a co-tenant on a split rental, who could watch a neighbor's GPU activity through them. To keep tenants isolated, Lium leases a profiling-enabled node **only as a whole host**:

- GPU splitting is suspended while the flag is on — every rental takes all GPUs;
- renters see the node in the **GPU Profiling (ncu)** filter only while it has no active rental.

Idle-time [default jobs](../portal/default-jobs.md) keep running while the node has no renter; the platform clears them before a renter's pod starts, as on any node.

Enable the flag if whole-node rentals fit your fleet. A node that mostly earns from 1×/2× split rentals loses that segment while the counters are open.

## Enable the counters

:::warning Only change the flag on an idle node
Opening the counters while rented pods are running exposes those renters to the side-channel above and trips a platform alert. Wait until the node has no active rentals (or drain it first).
:::

The commands below are for the Ubuntu host image Lium nodes ship with; on another distribution use its own initramfs tool.

1. Set the driver module option on the host:

```bash
echo 'options nvidia NVreg_RestrictProfilingToAdminUsers=0' | sudo tee /etc/modprobe.d/nvidia-profiling.conf
```

2. Rebuild the initramfs and reboot:

```bash
sudo update-initramfs -u
sudo reboot
```

3. Verify the loaded state after reboot:

```bash
grep RmProfilingAdminOnly /proc/driver/nvidia/params
```

Expected output:

```
RmProfilingAdminOnly: 0
```

The validator reads the loaded driver state on its next hardware scan of the node. Expect the **GPU Profiling** badge on [lium.io](https://lium.io) within an hour, not instantly.

Detection is fail-closed: if the platform cannot read the state (no NVIDIA driver, unreadable `/proc/driver/nvidia/params`), the node is treated as *not* profiling-capable.

## Disable the counters

Remove the option and reboot — again, only while the node has no active rentals:

```bash
sudo rm /etc/modprobe.d/nvidia-profiling.conf
sudo update-initramfs -u
sudo reboot
```

After the next hardware scan the badge disappears and the node returns to normal rental rules, including GPU splitting if configured.