Skip to main content

GPU profiling with ncu

Profile CUDA kernels with NVIDIA Nsight Compute (ncu) inside a Lium pod.

Why a special machine​

ncu reads GPU performance counters, and on a default driver configuration those counters are restricted to admin users on the host. Inside an ordinary pod, any profiling attempt fails with:

==ERROR== ERR_NVGPUCTRPERM - The user does not have permission to access NVIDIA GPU Performance Counters on the target device

Some providers open the counters on their machines (see the provider guide). Those machines carry a teal GPU Profiling badge in Browse Pods, and on them ncu reads the counters from an ordinary pod — no special container flags, just the CUDA toolkit (below).

Find one​

  1. Open Browse Pods on lium.io.
  2. Toggle GPU Profiling (ncu) in the filter rail. It shows machines where the counters are open and the whole machine is currently free.
  3. Rent as usual.

Because the counters are host-wide, a profiling machine is always rented as a whole host: the GPU-count selector is absent, every GPU goes to your pod, and nobody shares the machine while you hold it. The hourly price covers all GPUs — budget accordingly.

The exclusivity is also your protection: open counters would let a co-tenant observe your GPU activity, so Lium never co-schedules anyone with you on these machines.

Run ncu in the pod​

ncu ships with the CUDA toolkit. The default PyTorch template includes only the CUDA runtime, so install the toolkit first (or pick a devel-flavored template that bundles it). Pick the toolkit version the machine's driver supports — the Max CUDA driver row in the create-pod summary:

apt-get update && apt-get install -y cuda-toolkit-12-6

Then profile as you would on your own machine:

ncu --version                          # sanity check
ncu -o profile ./my_kernel_binary      # profile a binary, write profile.ncu-rep
ncu --set full -o profile python train.py

Copy the .ncu-rep report to your laptop (scp/rsync over the pod's SSH port) and open it in the Nsight Compute UI.

If ncu still prints ERR_NVGPUCTRPERM, you are on a machine without open counters — look for the GPU Profiling badge and move to a machine that has it.