Skip to main content

GPU Power Cap

Lium runs its own Default Jobs on your node while nobody is renting it — that is what the unrented incentive pays for. One of them caps GPU power before it starts, so the node container has to be allowed to run nvidia-smi -pl.

A node that cannot cap earns no idle payout

If the validator proves your node container cannot set a GPU power limit, the node keeps running: it stays active, verified and rentable, and its rental fees are untouched. Only the idle payout stops, and the reason cannot_apply_gpu_power_cap appears in the node's job log with the two readings behind the verdict; the node's day page in Earnings, under Incentive history, states the same verdict in plain words on its zero-paying cycles.

Check your node

Both readings come from inside the node container: CAP_SYS_ADMIN (bit 21 of CapEff) has to be held, and /dev/nvidiactl has to belong to root.

docker exec executor-executor-1 grep CapEff /proc/self/status
docker exec executor-executor-1 stat -c %u /dev/nvidiactl

Expected output on a healthy node:

CapEff:	000001ffffffffff
0

00000000a80425fb is the default Docker capability set: it has no CAP_SYS_ADMIN, and a node reading that loses the idle payout. An owner of 65534 means the device was mapped away from root.

Two notes on the container name. executor-executor-1 is the name on the standard stack — if you renamed the compose project, run docker ps and use your own name. Check the container whose name ends in -executor-1: the -monitor-1 container next to it runs the same image without those rights, so reading it reports a failure on a healthy node.

If either reading is missing or unreadable, the node is left alone — only a proven "cannot cap" counts.

Fixes

Pick the case that matches your setup; they are alternatives, not steps.

Restarting Docker kills running pods

Restarting the Docker daemon, or recreating the node container, stops every pod on the host. A pod killed under an active rental costs you a penalty — see Sysbox → applying changes. Drain the node first.

The standard stack, which stopped updating

The official node stack already starts the container with the rights it needs. The executor-executor-runner-1 container recreates the node container from its own compose file on every start, and executor-watchtower-1 keeps the runner current, so a node that is behind is usually a stack that stopped updating.

cd compute-subnet/neurons/executor
docker compose pull
docker compose up -d

Confirm afterwards that both executor-executor-runner-1 and executor-watchtower-1 are running — without Watchtower the stack never picks up new images on its own. To reinstall the node from scratch instead, use Node Quickstart → Step 2.

Your own compose file

If you start the node container from a compose file of your own (a renamed project, or a copy of the app compose kept outside the runner), grant the capability there:

services:
executor:
cap_add: ["SYS_ADMIN"] # or: privileged: true

Recreate the container afterwards with docker compose up -d --force-recreate, then re-run the check above.

A node container running under sysbox

An owner of 65534 on the device means the node container is running inside a sysbox user namespace, where the GPU device no longer belongs to root. The cap then fails even though the container holds every capability. This happens either because the container asks for runtime: sysbox-runc itself, or because the host's Docker default-runtime is sysbox-runc and every container inherits it.

Pods need sysbox; the node container does not. Lium asks for sysbox-runc per pod, explicitly, so keep it registered as a named runtime and not as the default one — confirm /etc/docker/daemon.json includes the sysbox-runc entry described in Sysbox, carries no "default-runtime": "sysbox-runc" line, and restart Docker. If your own compose sets runtime: sysbox-runc on the node container, drop that line too and recreate the container.

The idle payout resumes on the next validator cycle after any of these fixes — no portal action, no re-registration.