GPU Power Cap
Lium runs its own Default Jobs on your node while nobody is renting it — that is what the unrented incentive pays for. One of them caps GPU power before it starts, so the node container has to be allowed to run nvidia-smi -pl.
If the validator proves your node container cannot set a GPU power limit, the node keeps running: it stays active, verified and rentable, and its rental fees are untouched. Only the idle payout stops, and the reason cannot_apply_gpu_power_cap appears in the node's job log with the two readings behind the verdict; the node's day page in Earnings, under Incentive history, states the same verdict in plain words on its zero-paying cycles.
What runs on your GPUs while nobody rents them
When your node is idle, the load you see on the cards is Lium's own Default Job, not a stranger. It is the workload the unrented incentive pays for. On the host it runs as a container named filler_… in docker ps, and every start and stop appears under Node Activity on the node's page in the Provider Portal.
The heavy job (Pearl) holds the GPUs at full load, so Lium lowers the GPU power limit before it starts and restores the previous limit when the job stops. The light job (Dolphin) runs without a Lium cap. Which one runs on your node depends on the node's hardware and driver. A customer rental stops the job within seconds and runs at the limit set on the host.
If you do not want any Lium job on the node, assign a Miner Empty Job. The node stays rentable and earns no idle payout.
Set your own host power limit
You can lower the power limit on the host yourself, for example when the power supply or the cooling cannot hold two cards at full load. The validator accepts a host limit down to 90 % of the card's default limit. Below that, the check reports GPU_POWER_LIMIT_BELOW_DEFAULT and the node scores 0 (see Troubleshooting → Reason codes). Lium's own job cap is not counted against you. Before the validator caps for Pearl, it records the limit the card has at that moment, and it writes that value back when the job stops.
Renters see a lowered limit. A host limit under 95 % of the card's default puts a yellow Reduced power limit badge on the node in Browse Pods and on the pod's page, with your limit and the card's default in watts. While a Lium or miner default job runs on the node, a card whose limit reads at Pearl's 70 % figure or at the card's own minimum power limit is not judged and shows no badge: Pearl's cap clamps to that minimum on cards whose minimum is above 70 % of the default (409 W on an RTX 4090, 518 W on an RTX 5090), so a host limit set at the minimum cannot be told from Lium's cap. A host limit between the minimum and the default — 540 W on an RTX 5090, 630 W on an H100 — is judged under any job.
Read the default limit, then set yours:
nvidia-smi --query-gpu=name,power.default_limit --format=csv
sudo nvidia-smi -pm 1
sudo nvidia-smi -pl <watts>
Enable persistence mode (-pm 1) first. Without it, the driver unloads when the GPU goes idle and silently drops the limit. An RTX 5090 reports a 575 W default, so 518 W is the lowest limit that keeps the idle payout. Set the limit while no filler_… container is running (docker ps). A limit set while Pearl runs is replaced by the earlier limit when the job stops, so set it again then. Both settings reset on reboot, so put the two commands in a script that runs at boot, for example a systemd unit. A limit at or above the 90 % floor, set on an idle node, stays in place through Lium's jobs and through rentals. A limit below the floor is raised back to the default when a rental or Dolphin starts. The validator reads it on the next cycle, about 15 minutes later.
Do not put the limit back
Only the Pearl job gets a Lium cap. Dolphin runs without one. The cap lasts only as long as the Pearl job. Lium records your own limit before the job starts and writes it back when the job ends, so nothing is left capped on your host.
Some hosts run a script or a systemd timer that re-applies their own wattage every few seconds. That raises the limit under a running Pearl job. Lium sees the job running at full power, stops it, and the node then sits idle. The node did no work, so the payout paid for nothing.
While a Pearl job runs, the validator checks the live limit every cycle. It counts each job whose limit has moved above the cap Lium set and sits at 90% or more of the card default. If a card reports no default limit, the validator uses the limit it recorded on that card before the cap as the line instead. A whole node counts once, not once per GPU, and one job counts once however long it runs. After 2 such jobs in 24 hours the node keeps running — it stays active, verified and rentable, and its rental fees are untouched — but the idle payout stops. The reason reverts_gpu_power_cap appears in the node's job log with the capped and the found wattage, and in Incentive history on the node's day page in Earnings.
The rule is not live. Its code (lium-io#1249) is not released, and it ships in shadow mode: the validator counts the reverts and logs them, and the idle payout is paid as before. The payout stops, and reverts_gpu_power_cap appears in the job log, only once validators turn ENABLE_UNRENTED_POWER_CAP_REVERT_LIMIT on. Releasing lium-io#1249 alone changes nothing for your payout.
Only the limit DURING the job counts. If your script sets your own wattage back after the job ends, that is fine and costs you nothing. To fix a node that is losing the payout, stop whatever re-runs nvidia-smi -pl while a Pearl job is on the node. Your own crontab is one place it can hide. A systemd service that runs at boot, root's crontab, the system cron directories and /etc/rc.local are the others, so check all of them:
# systemd: timers, then services (a boot-time unit shows up here, not in list-timers)
systemctl list-timers --all
systemctl list-units --type=service --all | grep -iE 'nvidia|power|smi|gpu'
grep -rIl -- 'nvidia-smi' /etc/systemd/system /lib/systemd/system 2>/dev/null
systemctl cat <unit> # for each unit the two lines above list
# cron: your user, root, and the system directories
crontab -l
sudo crontab -l
ls /etc/cron.d /etc/cron.hourly /etc/cron.daily
sudo grep -rIl -- 'nvidia-smi' /etc/cron* /var/spool/cron 2>/dev/null
# boot script, if the host has one
[ -f /etc/rc.local ] && grep -n 'nvidia-smi' /etc/rc.local
A one-off change, for example a driver reload, is not enough to stop the payout. Each counted job expires on its own, 24 hours after the validator saw it. The payout resumes on the first validator cycle at which fewer than two counted jobs remain in the 24-hour window.
Check your node
Both readings come from inside the node container: CAP_SYS_ADMIN (bit 21 of CapEff) has to be held, and /dev/nvidiactl has to belong to root.
docker exec executor-executor-1 grep CapEff /proc/self/status
docker exec executor-executor-1 stat -c %u /dev/nvidiactl
Expected output on a healthy node:
CapEff: 000001ffffffffff
0
00000000a80425fb is the default Docker capability set: it has no CAP_SYS_ADMIN, and a node reading that loses the idle payout. An owner of 65534 means the device was mapped away from root.
Two notes on the container name. executor-executor-1 is the name on the standard stack — if you renamed the compose project, run docker ps and use your own name. Check the container whose name ends in -executor-1: the -monitor-1 container next to it runs the same image without those rights, so reading it reports a failure on a healthy node.
If either reading is missing or unreadable, the node is left alone — only a proven "cannot cap" counts.
Fixes
Pick the case that matches your setup; they are alternatives, not steps.
Restarting the Docker daemon stops every pod on the host. A pod killed under an active rental costs you a penalty — see Sysbox → applying changes. Drain the node first.
The standard stack, which stopped updating
The official node stack already starts the container with the rights it needs. The executor-executor-runner-1 container recreates the node container from its own compose file on every start, and executor-watchtower-1, the updater, keeps the runner current. A node that is behind is a stack that stopped updating. The validator reports it as EXECUTOR_IMAGE_OUTDATED ("Executor image is outdated") in the node's Job Logs and in the red last error panel on the node's page in the Provider Portal; Troubleshooting → Reason codes says what the verdict costs today.
What stopped the nodes: a mirror, not a pull limit
Until 18 September 2026 the standard compose file ran nickfedor/watchtower, which updates by tag. It asks a registry for the digest behind compute-subnet-executor-runner:latest and pulls only when that digest differs from the one on the host. On a host whose /etc/docker/daemon.json names a registry mirror, Watchtower and the Docker daemon ask that mirror first and take its first answer. A site mirror that kept an old copy of latest answered with the old digest, the digest equalled the one on the host, Watchtower logged Digest match, skipping pull, and Docker Hub was never asked. The node was not out of pulls. It never pulled.
The numbers behind this (lium-io#1359, read on 12 September 2026): 104 of 493 active nodes ran an old node image, 97 of them the runner that had been latest before the 11 September release, and 94 of the 104 sat in three network blocks. Whole sites behind one mirror, not single hosts, and not per-node egress.
The fix is in the stack (lium-io#1359, merged 18 September 2026). The compose file now runs daturaai/lium-watchtower, Lium's own updater (version 1.1.0 in the 18 September compose file; a later version does the same). At start, and then every five minutes, it reads the digest of the current runner image from lium.io, signed by the validator, and not from any registry. When the runner on the host runs a different digest, it pulls daturaai/compute-subnet-executor-runner@<digest> and recreates the runner. A pull by digest is content-addressed: the daemon checks the hash of what it receives, so a mirror can answer with the right image or with an error, never with an old one. When the daemon's path fails, the updater pulls the same digest from registry-1.docker.io directly, which the mirror setting does not cover. The new runner then starts the node container from an image pinned by digest too.
Move the node onto the updater, once
A stack installed before 18 September 2026 still runs nickfedor/watchtower, and nothing replaces it on its own: Watchtower never updated itself. Run this once on the host:
cd ~/compute-subnet/neurons/executor && git pull && docker compose pull watchtower && docker compose up -d watchtower
A lium mine installation puts the checkout at compute-subnet under the directory where lium mine was run, usually the home directory. For a manual installation, use the directory that holds neurons/executor/docker-compose.yml. git pull brings the compose file with the updater; the checkout must be on main (git branch --show-current). docker compose pull watchtower fetches the updater image and nothing else: a pull of the runner tag would return the mirror's old copy again, so the command does not ask for it. docker compose up -d watchtower replaces nickfedor/watchtower with the updater and leaves the runner alone. The updater checks at start, pulls the current runner by digest and recreates it, and the new runner recreates the node container. Pods are separate containers and keep running; the node container restarts once. The command can print a warning about orphan containers. The warning is harmless. Do not add --remove-orphans, because that flag removes the node containers. If git pull stops or a step fails, EXECUTOR_UPDATE.md in the node repository covers each case.
After about two minutes, confirm that both containers are up and that the updater is the new one:
cd ~/compute-subnet/neurons/executor && docker compose ps executor-runner watchtower
docker inspect executor-watchtower-1 --format '{{.Config.Image}}'
The first command lists executor-executor-runner-1 and executor-watchtower-1 with the state Up. The second prints daturaai/lium-watchtower:<version> (1.1.0 in the 18 September compose file; a newer version is fine); a host that still prints nickfedor/watchtower has not moved. Then make sure that the node runs the current runner. Both commands print a digest, and the two must be equal:
curl -s https://hub.docker.com/v2/repositories/daturaai/compute-subnet-executor-runner/tags/latest | python3 -c 'import sys, json; print(json.load(sys.stdin)["digest"])'
docker image inspect --format '{{index .RepoDigests 0}}' "$(docker inspect executor-executor-runner-1 --format '{{.Image}}')" | cut -d@ -f2
The first line reads Docker Hub's latest tag. The updater itself reads the validator-signed digest from https://lium.io/api/watchtower/digest (the read in EXECUTOR_UPDATE.md); the two are the same digest once a release is signed, and Docker Hub's can run ahead or behind it for the minutes around a release, so a difference right after a release is not a fault: check again a few minutes later.
When the updater itself has a new version
The updater keeps the runner current but does not update itself. When Lium announces an updater release, run the one-time command above (git pull && docker compose pull watchtower && docker compose up -d watchtower). Then check the installed version with docker inspect executor-watchtower-1 --format '{{.Config.Image}}'.
The next validator cycle after the node image matches the current release reports the node as current. What the EXECUTOR_IMAGE_OUTDATED verdict costs is in Troubleshooting → Reason codes: today it is a warning, and score and idle payout are unchanged (lium-io#1360); once validators turn EXECUTOR_IMAGE_CHECK_ENFORCE on, an outdated node scores 0 and earns no idle payout until its image is current.
If the digests differ after the restart, read the updater's log: docker logs executor-watchtower-1 --tail 50. Failed to fetch digest from endpoint means the host cannot reach lium.io; allow it in the outbound firewall. Signature verification failed means the clock is more than 10 minutes off (date -u), or the response did not come from the validator. Pull by digest failed through the daemon's registry path, retrying from Docker Hub directly means the mirror could not serve the digest and the updater went to Docker Hub itself; the next lines say whether that worked, and a toomanyrequests there is the case in the note below. EXECUTOR_UPDATE.md explains the other lines.
If the digests still differ, something between the host and Docker Hub serves an old copy of the tag. EXECUTOR_UPDATE.md says what to look at. To reinstall the node from scratch instead, use Node Quickstart → Step 2.
One case is a hold, not a fault. Read the updater's log with docker logs executor-watchtower-1 --tail 50. A line Update held: the NVIDIA runtime cannot start a GPU container on this host means Watchtower (daturaai/lium-watchtower 1.1.1 and later) found a new runner but did not install it, because the host cannot start a GPU container right now. A new node container would not start either, so the hold keeps the node and its pods running. Watchtower checks again every 5 minutes and installs the update on its own once the host is fixed. With "reason_code": "NVIDIA_RUNTIME_MISMATCH", the host's NVIDIA driver and its libraries are out of step, which happens after a driver package update without a reboot. The fix is a reboot or a driver reload, and both stop every running pod. First click Pause New Rentals on the node in the Provider Portal: the node then takes no new rental, and the pod that runs now is not touched. Wait until the rental on the node ends. Then reboot the host or reload the driver, and run docker compose up -d in the stack directory. With "reason_code": "RUNTIME_PROBE_FAILED", the error field says why, most often in the Docker daemon's words; EXECUTOR_UPDATE.md says how to reproduce the probe by hand and what the other lines in that log mean.
A different case: the host is out of Docker Hub pulls
This is not what stopped the nodes above. The nodes checked on 10 September 2026 had their full anonymous allowance left, and the mirror finding explained the stale sites. It is still a way for a node that runs the updater to lag, so it is kept here as a note.
Docker Hub allows 100 anonymous pulls per rate-limit window per source IPv4 address (per /64 on IPv6) at the time of writing (Docker Hub usage and limits). Every node behind the same egress shares one allowance, so a site with tens of nodes can run out even though each node pulls rarely. The updater's digest check reads lium.io, not Docker Hub, so it keeps seeing the new release and keeps failing the pull until the allowance frees up. While the site stays over the limit the node does not update. It updates on its own once the rate-limit window frees.
The pulls of a release go through the host's Docker daemon without a login: the updater pulls the runner image by digest, and the runner pulls the node image by digest. To see what they see, repeat the updater's pull the same way, by digest, through the daemon, with an empty Docker config so a docker login on the host cannot mask the containers' anonymous failure (the digest is read from Docker Hub's latest tag, which matches the signed digest the updater reads except for the minutes around a release):
docker info --format '{{.RegistryConfig.Mirrors}}'
digest=$(curl -s https://hub.docker.com/v2/repositories/daturaai/compute-subnet-executor-runner/tags/latest | python3 -c 'import sys, json; print(json.load(sys.stdin)["digest"])')
docker --config "$(mktemp -d)" pull "daturaai/compute-subnet-executor-runner@$digest"
toomanyrequests: You have reached your pull rate limit means the host has used up Docker Hub's anonymous pull allowance. A bare 429 Too Many Requests without that text is Docker Hub's abuse rate limit, which a login does not lift. If the first command printed a mirror ([] means none), the daemon asked that mirror first and went to Docker Hub only after the mirror failed to answer, so a toomanyrequests does not identify the cause on its own: it says that the mirror did not serve the image and that Docker Hub then refused this host's address, and the daemon does not print why the mirror failed. Check the mirror first: that it answers (curl -s <mirror URL>/v2/) and what its own log says. The updater's log shows the same fallback, Pull by digest failed through the daemon's registry path, retrying from Docker Hub directly, and the lines after it say whether the direct pull from registry-1.docker.io failed too.
A docker login on the host does not change what the containers pull: the pulls run inside executor-watchtower-1 and executor-executor-runner-1, and neither carries a login. What does help sits on the host side, because the host daemon makes both pulls. One fix is a pull-through cache shared by the site, set as registry-mirrors in /etc/docker/daemon.json on every node (Docker: mirror the Docker Hub library): one Docker Hub pull per release for the whole site. The runner and node images are pulled by digest, so the cache can only answer those with the right image or with an error, never with an old one. The one pull it can still answer with an old copy is the pull by tag on a fresh install (docker compose up -d pulls compute-subnet-executor-runner:latest); the updater corrects that at its first check. The other fix is fewer nodes per egress address: one public IP per node, or a NAT pool wide enough that no address serves more than a few nodes (Docker Hub usage and limits). A change to daemon.json takes a Docker restart, which stops every pod on the host, so drain the node first, as the caution above says.
Your own compose file
If you start the node container from a compose file of your own (a renamed project, or a copy of the app compose kept outside the runner), grant the capability there:
services:
executor:
cap_add: ["SYS_ADMIN"] # or: privileged: true
Recreate the container afterwards with docker compose up -d --force-recreate, then re-run the check above.
A node container running under sysbox
An owner of 65534 on the device means the node container is running inside a sysbox user namespace, where the GPU device is not owned by root. The cap then fails even though the container holds every capability. This happens either because the container asks for runtime: sysbox-runc itself, or because the host's Docker default-runtime is sysbox-runc and every container inherits it.
Pods need sysbox; the node container does not. Lium asks for sysbox-runc per pod, explicitly, so keep it registered as a named runtime and not as the default one — confirm /etc/docker/daemon.json includes the sysbox-runc entry described in Sysbox, carries no "default-runtime": "sysbox-runc" line, and restart Docker. If your own compose sets runtime: sysbox-runc on the node container, drop that line too and recreate the container.
The idle payout resumes on the next validator cycle after any of these fixes — no portal action, no re-registration.
The lowest limit the driver accepts
The Pearl Default Job asks the driver for 70 % of each card's default power limit. From thirty minutes after the launch, on every cycle, Lium checks the limits the validator last read on the run's GPUs. If a card still reads at or above 90 % of its default, the cap did not apply: Pearl stops on that node and does not come back for 12 hours.
Every card has a lowest power limit its driver accepts; nvidia-smi shows it as the min power limit. On some cards it sits close to the default: 409 W on a 450 W RTX 4090, 518 W on a 575 W RTX 5090. A request under that minimum settles at the minimum, so the card reads high even though the cap was applied as far as the driver allows.
Lium treats a limit at or under that minimum as honored: Pearl keeps running, no stop, no 12-hour pause. A reported minimum at or above the card's default is ignored, and the card is judged as before: a card whose minimum equals its default cannot be capped at all, and Lium does not accept a reading that would turn the check off for that card for good.
Nothing to do on your side. The node stays rentable, and its idle payout is not affected.