---
sidebar_position: 8
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lium.io/llms.txt
> Use this file to discover all available pages before exploring further.

# GPU profiling with ncu

Profile CUDA kernels with [NVIDIA Nsight Compute](https://developer.nvidia.com/nsight-compute) (`ncu`) inside a Lium pod.

## Why a special machine

`ncu` reads GPU performance counters, and on a default driver configuration those counters are restricted to admin users on the host. Inside an ordinary pod, any profiling attempt fails with:

```
==ERROR== ERR_NVGPUCTRPERM - The user does not have permission to access NVIDIA GPU Performance Counters on the target device
```

Some providers open the counters on their machines (see the [provider guide](../providers/nodes/gpu-profiling.md)). Those machines carry a teal **GPU Profiling** badge in **Browse Pods**, and on them `ncu` reads the counters from an ordinary pod — no special container flags, just the CUDA toolkit (below).

## Find one

1. Open **Browse Pods** on [lium.io](https://lium.io).
2. Toggle **GPU Profiling (ncu)** in the filter rail. It shows machines where the counters are open **and** the whole machine is currently free.
3. Rent as usual.

Because the counters are host-wide, a profiling machine is always rented **as a whole host**: the GPU-count selector is absent, every GPU goes to your pod, and nobody shares the machine while you hold it. The hourly price covers all GPUs — budget accordingly.

The exclusivity is also your protection: open counters would let a co-tenant observe your GPU activity, so Lium never co-schedules anyone with you on these machines.

## Run ncu in the pod

`ncu` ships with the CUDA toolkit. The default PyTorch template includes only the CUDA runtime, so install the toolkit first (or pick a `devel`-flavored template that bundles it). Pick the toolkit version the machine's driver supports — the **Max CUDA driver** row in the create-pod summary:

```bash
apt-get update && apt-get install -y cuda-toolkit-12-6
```

Then profile as you would on your own machine:

```bash
ncu --version                          # sanity check
ncu -o profile ./my_kernel_binary      # profile a binary, write profile.ncu-rep
ncu --set full -o profile python train.py
```

Copy the `.ncu-rep` report to your laptop (`scp`/`rsync` over the pod's SSH port) and open it in the Nsight Compute UI.

If `ncu` still prints `ERR_NVGPUCTRPERM`, you are on a machine without open counters — look for the **GPU Profiling** badge and move to a machine that has it.