Troubleshooting
When a node is online but earning nothing — or a freshly-installed node is failing validator verification — start here. This page covers the two diagnostic surfaces providers reach for first:
- Grafana Job Logs — the canonical place to read raw validator-side error messages for a specific node.
- The
verifyx-testbenchmark — runs the same network, storage, and memory checks the validator runs, on demand.
If neither narrows down the problem, the Provider Portal monitoring view and the full Grafana suite cover the rest of the day-2 telemetry.
A newly-added node stuck in VALIDATION_PENDING
If you just added the node and it sits at VALIDATION_PENDING with "Waiting for first validation from validator (usually within 15 min)", that is the pre-scoring state, not a failure. For the first 30 minutes the portal shows this line even when your miner logged a problem in the last 30 minutes. That problem is not a result for this node. Before a node can go AVAILABLE, a validator has to probe it once, and that first probe only happens after your account is connected to a coordinator:
- Central Provider Server (the default). The Central Miner Server switch in Profile Settings → Central Miner Configuration is ON for every new account (see Quickstart → Step 2); check that it still is. With the switch OFF and no miner of your own, nothing relays the validator's checks to your node, so it sits at
VALIDATION_PENDINGfor 30 minutes and then flips toNOT_DETECTED— with no error either way, because no check has run. - Self-hosted. Your own miner process must be running and connected — see Self-hosted provider.
Once connected, validators probe every 15 minutes, and the node is listed only after the validator has finished every provider in that cycle and published: Add Node → AVAILABLE takes 21 minutes at the median and 64 minutes at the 90th percentile (see Monitoring → What each status means). During that window VALIDATION_PENDING is expected: the probe has not run yet, so there is nothing to find in the logs below. After 30 minutes without a check of this node, the panel shows the newest miner-level cause from the last 30 minutes instead (see Monitoring → What each status means), or "last validation check didn't complete" when the portal does not know that cause. The portal only reads miner-level rows from the last 30 minutes: when the miner's problem was logged early and nothing newer followed, that row has aged out by then and the node reads NOT_DETECTED until a check of it runs. A real verification failure only shows up in the Job Logs once a probe has actually run; if the node has been VALIDATION_PENDING for more than an hour (past the 64-minute 90th percentile) with an opt-in already in place, start there.
1. Pull error details from Grafana Job Logs
The Job Logs dashboard is the per-node scoring log. Every validator run for your node — successful, failed, or unreachable — lands here with the exact error message the validator produced.
Open the dashboard with your filters
- Go to grafana.lium.io → Job Logs.
- In the top toolbar, set the two template variables:
miner_hotkey— your provider SS58 hotkey (the one registered on subnet 51).executor_id— the UUID of the specific node you're debugging. You can copy this from the Provider Portal node detail page or fromprovider.lium.io/executors/{executor-id}.
- Set the time range (top-right) to cover the period you care about. Default is the last 1 hour; widen it if your node has been silent.
You can also reach the same view without leaving the portal — each node's detail page has a Grafana tab that pre-fills both variables for that one node (see Monitoring Nodes → Grafana Integration).
Read the three log feeds
The dashboard shows three separate panels, all keyed by the same miner_hotkey + executor_id:
| Panel | When it has rows | What to look for |
|---|---|---|
| Job Logs | Every scoring run | The score the validator gave you, the uptime, and any non-fatal warnings. |
| Job Error Logs (Score = 0 AND GPU > 0) | The node was reachable but failed scoring | Synthetic-job errors, missing sysbox-runc runtime, GPU verification failures, network-metric failures. |
| Machine scrape Error Logs (Score = 0 AND GPU = 0) | The validator could not even read your machine | SSH / auth failures, unreachable host, no GPU detected, port closed. |
Triage rule of thumb:
- Rows in panel 3 → it is a connectivity or configuration problem (network, firewall, SSH, machine offline). Fix the host before doing anything else.
- Rows in panel 2 → the host is up but a specific check is failing. Read the error message; for verifyx network or storage failures, jump to §2 and reproduce locally.
- Rows in panel 1 with non-zero scores → your node is being scored normally; revenue concerns belong in Rewards and the rewards FAQ.
2. Run the verifyx benchmark to reproduce validator checks
When the Job Logs panel 2 reports a network or storage failure, the fastest way to confirm and iterate on a fix is to run the validator's own checks locally. We ship them as a Docker image, daturaai/verifyx-test, that runs on the node host itself. Run it before registering a new node, and re-run it any time validator-side network or storage scoring drops.
Quick start (network only)
docker run --gpus all --rm daturaai/verifyx-test:latest
Runs a 10-iteration network benchmark and prints download and upload speeds. Finishes in roughly 2–3 minutes.
Full benchmark (network + storage + memory)
docker run --gpus all --rm daturaai/verifyx-test:latest python3 benchmark.py --full
Runs all three test suites. Expect 5–10 minutes due to the 5 GB storage I/O test.
What is tested
| Test | What it measures | Minimum requirement |
|---|---|---|
| Network download | PyPI package download speed | 100 Mbps as an EMA across cycles (MIN_VERIFYX_EMA_DOWNLOAD_SPEED_MBPS in the validator). The first cycle's sample is the EMA, so one cold measurement under 100 Mbps fails the first cycle; a node that then measures faster usually passes the next one. The validator repeats the check every 15 minutes. There is no button to press. |
| Network upload | Cloudflare speedtest upload | — |
| Storage write | Sequential write throughput (5 GB) | 100 GB free space |
| Storage read | Sequential read throughput (5 GB) | — |
| Memory allocation | Allocates 75 % of available RAM (8–128 GB range) | 8 GB RAM |
Interpreting results
The benchmark runs 10 iterations and reports an Exponential Moving Average (EMA) with alpha = 0.3 (decay = 0.7), so recent runs carry more weight. The final printed values are the EMA-smoothed numbers — a single slow run will not dominate the result.
The last section of the output also shows your hardware stats (total RAM, free disk space, utilization) from the most recent successful run.
Prerequisites
Your host must have the NVIDIA Container Toolkit installed so Docker can pass GPU access through the --gpus all flag. If you followed the Node setup guide, this is already configured.
# Verify the toolkit is installed
docker run --gpus all --rm nvidia/cuda:12.8.1-base-ubuntu22.04 nvidia-smi
3. Common issues
| Symptom (Job Logs message or benchmark output) | Likely cause | Fix |
|---|---|---|
VERIFYX_FAILED_NETWORK_SPEED_TOO_SLOW — EMA download speed below 100 Mbps | ISP throttling or congested uplink; on a brand-new node, a single cold first sample | Re-run the benchmark above; if it is consistently under 100 Mbps, contact your hosting provider or move to a dedicated port. A first-cycle failure on a node that measures well locally clears on the next cycle. The validator repeats the check every 15 minutes. There is no button to press. |
| Storage test fails — not enough free space | Less than 100 GB free on the node disk | Free space or resize the volume. See Docker storage. |
| Memory allocation fails | Less than 8 GB RAM available to Docker | Reduce other processes consuming RAM; upgrade RAM if needed. |
docker logs executor-watchtower-1 says Update held: the NVIDIA runtime cannot start a GPU container on this host | The host cannot start a GPU container, most often because the NVIDIA driver and its libraries are out of step after a driver package update without a reboot ("reason_code": "NVIDIA_RUNTIME_MISMATCH") | Not a fault: Watchtower holds the runner update so the node and its pods stay up, and installs it on its own once the host is fixed. First click Pause New Rentals on the node in the Provider Portal: the node then takes no new rental, and the pod that runs now is not touched. Wait until the rental on the node ends, then reboot the host or reload the driver, then docker compose up -d. See The standard stack, which stopped updating. |
--gpus all flag not recognized | NVIDIA Container Toolkit not installed | Follow the Node setup guide. |
Renters see no pod logs or GPU metrics for pods on this node; the node container's log (docker logs executor-executor-1) says Refusing a signed /containers request: Signed timestamp is …s behind the executor clock; the accepted window is 300s (or ahead of) | The node's clock is more than 5 minutes off. The platform signs each pod-log and pod-metrics request with the time it was made, and the node refuses one outside the window. | Sync the clock: timedatectl set-ntp true, then check timedatectl shows System clock synchronized: yes. The window is 5 minutes either way and is fixed; correct the clock. |
| Job Logs panel 3 has rows; SSH / auth errors | Validator cannot reach the node | Check firewall, the node's public port, SSH service, and the credentials registered in the Provider Portal. |
SSH checks fail in runs with Not allowed at this time or Connection lost while the node is up and its port is open; a rented node is later marked inactive mid-rental | The node's SSH port sits behind a NAT (CVM, or Docker's userland proxy), so every connection arrives from one source address, and the node image's OpenSSH 10 penalised that address after scanners' failed logins | Update the node image: on the standard stack executor-watchtower-1 does it (stopped updating? see GPU power cap → Fixes); on a CVM redeploy with the new runner digest. The fix is in executor image v1.124 and later. Self-check: `docker exec executor-executor-1 sshd -T |
Job Logs panel 2 mentions sysbox-runc | Sysbox runtime missing or not the default | Install Sysbox — see Sysbox setup. It is required; validators reject any node without it. |
Job Logs mention Advertised CPU cores exceed the host's physical cores | The advertised CPU(s) count exceeds the kernel-present CPU population (/sys/devices/system/cpu/present) by more than 4 | Advertise the node's real core count. Until then, the node scores 0 in both the rented and the unrented pool. Read both counts yourself — see §5. |
Job Logs mention gpu requires reset, unknown device, nvml error: unknown error, or a driver/library version mismatch | The GPU fell off the bus, or the driver is half-upgraded | The node is quarantined — see §4. |
Job Logs mention cannot_apply_gpu_power_cap | The node container is not allowed to set a GPU power limit | The node stays rentable but earns no idle payout — see GPU power cap. |
Job Logs mention reverts_gpu_power_cap (not released yet; withholds nothing until validators turn it on) | A script or timer on the host put the GPU power limit back while Lium's Pearl job was running, on two jobs within 24 hours | The node stays rentable but earns no idle payout until fewer than two counted jobs remain in the 24-hour window. Stop whatever re-runs nvidia-smi -pl under a Pearl job — see Do not put the limit back. |
Job Logs mention duplicate_executor_in_cycle (not released yet) | The same executor was reported under two or more miner hotkeys in one cycle | The copy under the lowest hotkey is paid; the others earn 0 and show this line in their Job Logs. List each executor under one miner hotkey, once. |
Job Logs mention Executor image is outdated (EXECUTOR_IMAGE_OUTDATED) | The node runs an older daturaai/compute-subnet-executor image | A warning: the node keeps its score and its idle payout while auto-update is being repaired for the fleet. One exception applies now: if a cycle that logs this warning gives an unrented node no positive score, the node leaves the market. Once validators enforce the image check, an idle node on an old image scores 0 and leaves the market, and a rented node keeps its rental but earns no incentive (Job Logs then also mention outdated_executor_image). Update the node either way — see The standard stack, which stopped updating. |
Node stays VALIDATION_PENDING right after Add Node; no rows in any Job Logs panel yet | The account is not connected to a coordinator, so no validator has probed the node yet | Opt into the Central Provider Server (or start your self-hosted miner); validators probe every 15 minutes; the listing follows 21 minutes at the median, 64 at the 90th percentile, after you connect. See A newly-added node stuck in VALIDATION_PENDING. |
4. Quarantine: your node was pulled off the market
A fatal GPU-runtime fault, a node that cannot look up a public image registry, or a run of rentals that
fail to start for reasons on the host's side does not score the node down; it quarantines it: the node
scores 0 and receives no new rentals until it passes a clean verification.
| Verdict | What the host reported | What fixes it |
|---|---|---|
GPU_RUNTIME_DEVICE_FAULT | gpu requires reset, unknown device, or nvml error: unknown error | Reset the host. Reboot it, then check nvidia-smi and dmesg for an Xid before putting it back. A driver reconcile does not help here. |
GPU_RUNTIME_NVML_MISMATCH | driver/library version mismatch | Reconcile the driver — the loaded kernel module and the userspace NVML library are different versions. |
REGISTRY_DNS_UNRESOLVABLE | a renter's image pull from Docker Hub, GHCR, NGC (nvcr.io), Quay, GCR, registry.k8s.io or ECR Public failed with dial tcp: lookup <registry> …: i/o timeout (or server misbehaving, connection refused, no such host, Temporary failure in name resolution), or a Docker Hub pull timed out (or was refused) looking up the registry mirror set in /etc/docker/daemon.json | Fix the host's DNS. The node's resolver in /etc/resolv.conf, or the mirror, does not answer, so images that are not already on the node fail or stall on pull. Remove a mirror that does not answer; docker pull of an image the node does not have must start at once. The node leaves the market only while it has no renter; the next clean check lists it again. |
GPU_RUNTIME_MISSING_RENTED_GPU | GPU 'GPU-…' requested by tenant not present on executor | The node's cards changed since the validator last read them (a recreated cloud VM, a swapped card), so a pod asked for a GPU the host does not have. Re-register the node to list its current cards: a node's GPU set is fixed at its first check, and a different set fails every later check as GPU_UUID_CHANGED (§6), so the quarantine does not lift on its own. If the original card comes back, the next clean check lists the node again. |
CREATION_FAILURE_RUN | "N rentals in a row failed to start on this node", with no rental in between that got a container. Counted: a failed step that runs whatever image the renter chose (SSH to the node, clearing old containers, preparing volumes, GPU flags, the power cap, the port check), a pull or start of an unmodified Lium image, or a node the miner could not reach at all. Not counted: a renter's own image (a bad tag, a private registry, a command that exits), a registry that rate-limits, and default jobs. The detail page names the step that failed. | Check Docker on the host. docker ps shows the executor container up, and its logs (docker logs executor-executor-1) show no errors; docker info answers; df -h shows free disk for images and volumes; docker run --rm --gpus all <image> nvidia-smi -L starts the default rental image (recommended_image under RECOMMENDED_IMAGE_NOT_CACHED); the renting port range is open. A node another renter is on stays listed. The next clean check lists the node again. |
Getting back. There is no button to press and no ticket to open: once the fault is gone, the next clean check marks the node active again, roughly one cycle (~15 minutes) later. If it is quarantined again straight away, the fault is still there — read the exact stderr in the Job Logs panels.
A driver version marked invalid
A driver version the platform cannot match to a binary NVIDIA publishes is marked invalid, and every
node running it scores 0. The mark is temporary: it is re-decided about a day later, so a
version branded by mistake clears on its own — no ticket needed. If yours does not clear, the
platform still cannot verify that version; install a driver release NVIDIA publishes.
5. Check your node's CPU count
Read both numbers where the validator reads them — inside the node container:
docker exec executor-executor-1 lscpu | grep '^CPU(s):'
docker exec executor-executor-1 cat /sys/devices/system/cpu/present
executor-executor-1 is the container name on the standard stack; run docker ps if you renamed
the compose project. The second command prints CPU numbers, not a count: 0-127 means 128 CPUs, and
commas mean several ranges — 0-19,40-59 is 40. An honest host matches, even one with cores
offlined.
Only an advertised count more than 4 above the kernel number fails. A lower count never does,
and this check alone never zeroes a node it could not measure: an unreadable present file, or a
scrape with no core count in it, passes.
6. Validator reason codes
Every verdict the validator writes carries a reason_code. It is the title of the red last error panel on the node's detail page in the Provider Portal (see Monitoring → The last error panel) and the reason_code field in the JSON after >>> in the Job Logs. The codes below are the ones that score a node 0 or withhold its incentive (EXECUTOR_IMAGE_OUTDATED is listed ahead of enforcement: today it is a warning), listed most common first. Codes with severity info are not listed: they do not change the score. The one exception is GPU_VERIFY_SKIPPED_RENTED, which is in the table because it replaces a GPU_VERIFY_FAILED a provider would otherwise ask about.
Before you open a ticket
The panel already carries what support would ask you for. Work through this list first; most verdicts clear without a ticket.
- Read the reason code (the panel title) and the How to fix line, and find the code in the table below — the row's What to do column is the fix.
- Open Technical details (
what_we_saw). ForGPU_VERIFY_FAILEDthestderrthere names the CUDA error; match it in WhichGPU_VERIFY_FAILEDcase is yours and run that case's commands. ForVERIFYX_FAILED_NETWORK_SPEED_TOO_SLOWsee §2; forGPU_RUNTIME_*see §4; forCPU_QUOTA_EXCEEDS_HOST/CPU_TRUTH_MISMATCHsee §5; forRENTAL_PROBE_FAILEDthefailed_stepthere names the step, see §7. - Apply the fix, then wait one verification cycle (about 15 minutes). There is no button to re-run a check; the next clean cycle restores the node.
- Still failing after a clean cycle? Open a ticket through the support flow with three things: the reason code, the
what_we_sawJSON (copy it from the panel), and the output of the commands you ran. That is everything the first reply would otherwise ask for.
Reason code table
| Reason code | What the validator saw | Effect | What to do |
|---|---|---|---|
GPU_VERIFY_FAILED | The matrix-multiplication probe the validator runs inside a container on your node did not produce the expected result. Technical details → stderr carries the CUDA error the probe hit. | Score 0. | Match the stderr line in Which GPU_VERIFY_FAILED case is yours — CUDA cannot start (fabric), no CUDA device in the container, or out of memory. If none matches: docker run --rm --runtime=sysbox-runc --gpus all daturaai/compute-subnet-executor:latest nvidia-smi must list every GPU; look for leftover containers or host processes holding the cards (nvidia-smi), then wait one cycle. |
SYSBOX_REQUIRED_MISSING | The sysbox-runc runtime is not installed or not usable on an unrented node, or the validator's Docker-in-Docker probe container started but never answered on SSH. | Score 0 for the cycle; repeated failures deactivate the node. | Read the event's dind_probe_error first. DIND_INNER_DOCKERD_IPTABLES means the probe's inner dockerd could not use legacy iptables on your host: run sudo modprobe -a ip_tables iptable_nat iptable_filter (the installer's --check prints it) — reinstalling Sysbox does not help. No dind_probe_error: install Sysbox and verify with the command on that page. |
NVIDIA_RUNTIME_MISMATCH | The NVIDIA container hook refused the container of the validator's Docker-in-Docker probe on an unrented node: the host's kernel driver and its user-space NVIDIA library are different versions, most often after a driver package update without a reboot. Technical details → dind_probe_error carries the hook's own line (nvml error: driver/library version mismatch). The host can start no GPU container in this state, so the cycle measured nothing about Sysbox. The executor updater holds updates on the same host condition under this code — see the Update held row in §3. | Score 0 for the cycle; repeated failures deactivate the node. | Reboot the host, or reload the driver. Then run docker run --gpus all --rm nvidia/cuda:12.8.1-base-ubuntu22.04 nvidia-smi on the host; if the hook still refuses, reinstall the NVIDIA Container Toolkit (Node setup guide). The sysbox check runs again on the next cycle. |
NVIDIA_CONTAINER_HOOK_FAILED | The NVIDIA container hook refused the probe's container because NVML failed on the host; Technical details → dind_probe_error carries the hook's own nvml error: line. The host can start no GPU container in this state, so the cycle measured nothing about Sysbox. | Score 0 for the cycle; repeated failures deactivate the node. | Make nvidia-smi work on the host first (reset the GPU with nvidia-smi -r when nothing runs on it, or reboot), then run docker run --gpus all --rm nvidia/cuda:12.8.1-base-ubuntu22.04 nvidia-smi; if the hook still refuses, reinstall the NVIDIA Container Toolkit (Node setup guide). The sysbox check runs again on the next cycle. |
VERIFYX_FAILED_NETWORK_SPEED_TOO_SLOW | The EMA of the VerifyX download measurement is under 100 Mbps. Technical details shows ema_verifyx_download_speed and min_download_speed_mbps. | Score 0. | See §2. A first-cycle failure on a new node that measures well locally clears on the next cycle. |
SCRAPE_FAILED / SCRAPE_PARSE_FAILED | The machine-spec scrape the validator runs over SSH in the executor container failed to run, or returned output it failed to decrypt or parse. Newer validators report a scrape that failed to run under one of the five SCRAPE_* codes below: three say what the host answered, two say the validator got no answer back. | Validation halted, GPU unverified, score 0. | docker logs executor-executor-1 for the traceback; make sure the container is up to date (docker compose pull && docker compose up -d in neurons/executor) and that the host has free disk and memory. |
SCRAPE_FAILED_NO_GPU | The scrape ran on the host and the NVIDIA driver listed zero GPUs. | Validation halted, GPU unverified, score 0. | Run nvidia-smi on the host, then docker exec executor-executor-1 nvidia-smi: both must list every GPU. If only the host lists them, the container is starting without GPU access — check the NVIDIA Container Toolkit and the deploy.resources.reservations.devices block (driver nvidia, count: all) in neurons/executor/docker-compose.app.yml. |
SCRAPE_FAILED_DRIVER | The scrape ran on the host and the NVIDIA driver returned an error: the driver is unloaded, a GPU dropped off the bus, or the driver and its library are at different versions. Technical details → gpu_scrape_error carries the driver's error. | Validation halted, GPU unverified, score 0. | nvidia-smi on the host and dmesg for NVRM: Xid lines. A reboot or a driver reinstall stops every pod on the node: if it is rented, first click Pause New Rentals in the Provider Portal and wait until the rental ends. Then reboot the host, or reinstall the driver when the versions differ. |
SCRAPE_FAILED_ON_HOST | The scrape ran on the host and exited with an error (a traceback, a missing library, a killed process) or finished with empty output. | Validation halted, GPU unverified, score 0. | docker logs executor-executor-1 for the traceback; update the container (docker compose pull && docker compose up -d in neurons/executor) and make sure the host has free disk and memory. |
SCRAPE_TIMEOUT | The validator stopped waiting for the scrape at its timeout. Either side can cause this: a GPU query that hangs on the host, a host link that drops packets, or a busy validator. | Validation halted, GPU unverified, score 0. | A single timeout clears when the next cycle passes. If it repeats: nvidia-smi on the host must answer within a few seconds (a query that hangs points at the driver or a card — dmesg for NVRM: Xid lines); mtr or ping to the node's IP from another machine shows whether its link drops packets; df -h and free -h on the host show free disk and memory. |
SCRAPE_TRANSPORT_FAILED | The SSH session ended before the host sent the scrape's exit status, or the upload of the scrape to the node failed. Either side can cause this: the host's network, sshd or the executor container going down, a full host disk during the upload, or the validator's own network. Technical details → error_type (or fallback_upload_error when the upload failed) names what the validator saw. | Validation halted, GPU unverified, score 0. | The next cycle retries. If it fails on every cycle: the SSH port (2200 by default) must answer from outside the host (nc -vz <node-ip> 2200 from another machine); docker ps must show the executor container up, and docker logs executor-executor-1 shows any restarts; df -h must show free disk on the host. |
INSUFFICIENT_PORTS | Fewer than 3 of the ports in the node's renting port range are reachable from the validator. | Score 0. | Open the renting port range you configured in lium mine on the firewall / NAT, or widen it in neurons/executor/.env and restart the container. |
UPLOAD_FAILED | The validator could not copy its check scripts onto the node over SSH. | Validation halted, score 0. | Check free disk space on the host, that the SSH port (2200 by default) is reachable from outside, and docker logs executor-executor-1 for SSH errors. |
GPU_USAGE_HIGH / GPU_USAGE_OUTSIDE_TENANT / GPU_FOREIGN_PROCESS / GPU_VRAM_HELD | A GPU on an idle node is busy, holds VRAM, or runs a process that is not a Lium pod or default-job container — another miner, a host-side job, or a hidden process. | Score 0. | Stop every non-Lium GPU workload on the host until nvidia-smi shows no processes and no used memory. Capacity sold on Lium must be free for the renter. |
ORPHANED_RENTAL_CONTAINER | A rental ended but its container is still running on the host. | Score 0. | docker stop <container> — the name is in Technical details. |
EXECUTOR_TRANSPORT_UNREACHABLE / FILLER_TRANSPORT_UNREACHABLE | An SSH connection to the node dropped mid-check (network blip, connection-tracking flush). | No verdict for that cycle. | Nothing if it happens once. If it repeats, check the host's network path and the SSH port. |
VERIFYX_FAILED (with a suffix: _SSH_TRANSPORT, _EXECUTOR_CRASH, _EMPTY_RESPONSE, _CIPHER_REJECTED) | The VerifyX hardware probe failed for a reason other than speed: SSH dropped, the probe process crashed on the node, returned nothing, or returned a response the validator rejected. | Score 0. | Read docker logs executor-executor-1 for the traceback; dmesg for OOM kills and df -h for a full disk; restart the container so it runs the current image. The validator's own debugging notes are in verifyx-debug.md. |
FILLER_STOPPED_BY_HOST / FILLER_CONTAINER_MISSING / FILLER_KILLED_AT_CREATE | Something on the host stopped or removed a Lium default-job (filler_*) container while its run was active — repeatedly, in the _AT_CREATE case. | Unrented incentive withheld until a default job runs undisturbed. | Do not stop filler_* containers; allow-list them in any host-side cleanup tool (Watchtower, cron docker prune, another subnet's agent). See Default Jobs. |
RENTAL_NOT_VERIFIED | A rented pod's container could not be verified on the node. | Validation halted. | Check the pod's container is running (docker ps); if it is gone but the pod still shows as rented, see Force-closing a pod. |
POD_NOT_RUNNING | A pod the backend records as running is not running on the node. | Score 0, verification cleared; a penalty is proposed and confirmed two hours later unless a later check passes. | Same as above. When the validator sends it, the penalty event records what it could read from the container (state, exit code, whether it was OOM-killed — or, when docker inspect failed on the host, only Docker's error and whether the container is gone) — see What a penalty event records. |
PROVIDER_BANNED / GPU_BANNED | Your hotkey, coldkey or a GPU UUID is on the ban list. | Score 0; running rentals finish. | See Provider ban. |
EXECUTOR_IMAGE_OUTDATED | The executor container is not running the current image. | A warning; score and idle payout unchanged. One exception applies now: if a cycle that logs this warning gives an unrented node no positive score, the node leaves the market. Once validators enforce the image check: score 0 until the image is current. | Run the one-time command in The standard stack, which stopped updating: cd ~/compute-subnet/neurons/executor && git pull && docker compose pull watchtower && docker compose up -d watchtower. A plain docker compose pull && docker compose up -d keeps the old updater and pulls the runner tag through the same mirror that served the old copy. Or re-run lium mine. |
GPU_VERIFY_TIMEOUT | The matrix probe did not finish in time. | Score 0. | nvidia-smi for stuck processes or containers holding VRAM; reboot the host if a GPU is wedged. |
RECOMMENDED_IMAGE_NOT_CACHED | The default rental image for your GPU and driver is not on the host. Technical details names it as recommended_image, and prefetch_state says what the node's own pre-pull last did. | Score 0. | If prefetch_state names a pull error or low disk, fix that first. Then docker pull <recommended_image> on the host and wait one cycle. docker logs executor-executor-1 has the pre-pull's own messages. |
GPU_VERIFY_VRAM_UNAVAILABLE | The matrix probe could not allocate GPU memory: another process held VRAM while it ran. Older validators report this as GPU_VERIFY_FAILED with "UUID mismatch … got 'None'". | Score 0. | nvidia-smi for processes or leftover containers on the GPU; the next cycle retries. Details: Out of memory. |
GPU_VERIFY_SKIPPED_RENTED | The matrix probe could not get the GPU because a Lium pod or default-job (filler_*) container started on your node after the cycle's rental snapshot was taken. The validator confirmed that container with the backend before it wrote this. Technical details → workload says which kind, containers names them, and probe carries the probe's own output. | Informational. Score unchanged; the next cycle verifies again. | Nothing. |
GPU_VERIFY_CUDA_NOT_READY | CUDA cannot start on the GPU although nvidia-smi works (CUDA error 802, system not yet initialized); on HGX H100/H200 boards the NVLink fabric is not ready. Older validators report this as GPU_VERIFY_FAILED. | Score 0. | Start Fabric Manager, have the hosting provider activate the fabric partition, or reinstall the driver on a board without NVSwitch — which one applies. |
GPU_VERIFY_NO_CUDA_DEVICE | The probe found no usable CUDA device inside the node container: the NVIDIA Container Toolkit is not exposing the GPUs, or the host driver is older than the probe's CUDA runtime. Older validators report this as GPU_VERIFY_FAILED. | Score 0. | docker exec executor-executor-1 nvidia-smi must list every GPU — the four fixes. |
GPU_FAULT_PROBE_FAILED (behind the validator flag GPU_FAULT_PROBE_CHECK_ENABLED, off today) | The kernel-fault probe that follows the matrix probe (random gather/scatter, atomics, a pointer chase and a copy round-trip over ~2 GB of VRAM) hit a CUDA error, returned wrong data, hung, or NVML reported new uncorrected ECC errors / remapped rows. Technical details names the GPU index and the error. | Warning while the probe is in shadow mode; score 0 once the validator enforces it. | nvidia-smi -q -d ECC,ROW_REMAPPER and the host dmesg for NVRM: Xid lines. Reset the card (nvidia-smi -r, with nothing running on it) and wait one cycle; a card that fails again needs reseating or replacing. |
RENTAL_PROBE_FAILED (off today; the team turns the probe on per validator) | The validator rented your idle node the way a customer does and one step failed: container_start, sshd_listen, ssh_login or gpu_count. Technical details carries failed_step, the image it used, ssh_port once the container existed, and the time each step took in probe_steps. | Score 0 for the cycle, verification cleared, so the node is off the market for that cycle. | Match failed_step in §7 and run that step's command. The next cycle probes the node again while it stays idle. |
GPU_UUID_CHANGED | The node shows a GPU that is not in the set recorded on its first successful check. Causes: a swapped card, or a cloud VM placed on another host. The recorded set is kept, so the node does not pass again on its own. An added card is reported here too, and so is a missing card while the anchor rule (GPU_ANCHOR_HARD_ENABLED, off today) is off. | Score 0 on this and every later cycle. A missing card: score 0 until the full set is back. With the anchor rule on, the event says GPU set differs from the listed set and the node is marked broken. | To list a different set of GPUs, delete the node in the portal and register it again. Uptime starts at 0; penalties stay. See One node, one fixed set of GPUs. |
GPU_MISSING (behind the validator flag GPU_ANCHOR_HARD_ENABLED, off today) | One or more GPUs of the recorded set are not visible. While the flag is off this case shows as GPU_UUID_CHANGED. | Verification reset, score 0 until every recorded GPU is visible again. | Check nvidia-smi -L and dmesg on the host; a card dropped off the bus. The node scores again when the full set is back. |
SPEC_CHANGED | The GPU model or count differs from the recorded model:count while the GPU UUIDs still match (a missing or added card is GPU_UUID_CHANGED). | Verification reset, score 0 on every cycle until the model and count match the recorded ones again. | A changed model: re-register the node. |
GPU_POWER_LIMIT_BELOW_DEFAULT | A GPU's power limit is under 90 % of its NVML default. | Score 0. | nvidia-smi -pl <default> to restore it; see GPU power cap for how Lium's own filler caps are told apart. |
GPU_DETAILS_MISMATCH / GPU_COUNT_ZERO / GPU_VRAM_MISMATCH / GPU_COUNT_EXCEEDS_MAX / GPU_MODEL_UNSUPPORTED | The GPU count, model or VRAM the node reports is inconsistent, zero, above the per-node maximum, or not on the supported list. | Score 0. | Make the registered GPU type and count match nvidia-smi -L; do not modify what the executor reports. |
GPU_MODEL_MIXED | The node's GPUs are more than one model. Every GPU of a node must be the same model. | Score 0; the node is not listed and cannot be rented. | Move the odd card to its own node, or run the node with cards of one model only. |
NVML_DIGEST_MISMATCH / NVML_DRIVER_UNKNOWN | The NVIDIA driver library on the node does not match a known build, or the driver version is not one the platform recognizes. | Score 0; previous verification cleared. | Reinstall a driver release NVIDIA publishes; see A driver version marked invalid. |
GPU_RUNTIME_DEVICE_FAULT / GPU_RUNTIME_NVML_MISMATCH / GPU_RUNTIME_MISSING_RENTED_GPU / GPU_WEDGED | The GPU is not addressable, the driver is half-upgraded, a pod asked for a card that is missing from the node, or a card is pinned busy with no process. | Quarantined: no new rentals until a clean check. | §4. |
CPU_QUOTA_EXCEEDS_HOST / CPU_TRUTH_MISMATCH | The advertised CPU count is above what the host has. | Score 0. | §5. |
PROVIDER_SIDE_LOAD_ABOVE_LIMIT | A workload outside Lium containers is using the node's CPU or disk. | Score 0. | Stop it. |
PORT_VERIFY_FAILED | The validator could not reach a port it mapped on the node. | Score 0. | Check Docker port mappings and the firewall for the renting port range. |
RENTED_EXECUTOR_NOT_LISTED (behind a validator flag, off today) | The node is rented, and your miner's answer to the validator's cycle did not include it. The validator writes this row so the platform sees the outage. Only that is known: why the miner left the node out is not in the row. | Score 0 for the cycle; the node is hidden from the market until a cycle lists it again. For the mid-rental inactivity check the row weighs as the silence it replaces: no more, no less. | Check that the node is up and that your miner lists it (the executor container running, its executor port answering the miner). With the Central Miner Server switch on, the node must answer Lium's miner. |
EXECUTOR_DUPLICATE | Two registered nodes share one executor UUID. | Score 0, verification cleared. | Deregister the duplicate in the portal. |
INSPECTOR_* | The tenant-workload inspector could not run or flagged something. | Score unchanged. | Informational; if it repeats every cycle, restart the executor container so it ships the current libinspector. |
*_CONFIG_MISSING, MISSING_REMOTE_DIR, *_POLICY_MISSING | The validator's own configuration is incomplete. | Validation halted for the cycle. | Nothing on your side. If a node shows one of these for more than two cycles, report it through the support flow. |
Which GPU_VERIFY_FAILED case is yours
The probe allocates GPU memory and runs a matrix multiplication inside the node container. When CUDA refuses, the verifier prints one line ending in the CUDA error text — Failed to allocate d_A: <CUDA error> for an allocation, cudaGetDeviceCount returned <CUDA error> when it finds no device — and that text, in the panel's Technical details → stderr or in the Job Logs JSON, tells the cases apart. nvidia-smi working on the host proves only that the driver is loaded; each case below is a way CUDA can still fail to start a computation.
Today Learn more opens the top of §6; scroll down to this section. Each case is its own reason code — GPU_VERIFY_CUDA_NOT_READY, GPU_VERIFY_NO_CUDA_DEVICE, GPU_VERIFY_VRAM_UNAVAILABLE — with its row in the table above and the panel's How to fix line naming the fix. Older validators still report GPU_VERIFY_FAILED; for those, match the stderr yourself.
What stderr says | Cause | Commands that tell you which fix applies |
|---|---|---|
system not yet initialized (CUDA error 802, cudaErrorSystemNotReady) | CUDA cannot start on the GPU although nvidia-smi works. On HGX H100/H200 boards the NVLink fabric is not ready. | nvidia-smi -q | grep -A4 -i fabric · systemctl status nvidia-fabricmanager · systemd-detect-virt · nvidia-smi --query-gpu=name,driver_version --format=csv — see CUDA cannot start. |
cudaGetDeviceCount returned no CUDA-capable device is detected (cudaErrorNoDevice, CUDA_ERROR_NO_DEVICE), or CUDA driver version is insufficient for CUDA runtime version (cudaErrorInsufficientDriver — the probe's CUDA runtime found no usable driver library in the container) | The node container cannot use a GPU: the container runtime or the NVIDIA Container Toolkit is not exposing the cards to it, or the host driver is older than the probe's CUDA runtime. | docker exec executor-executor-1 nvidia-smi must list every GPU; if Docker itself answers could not select device driver "" with capabilities: [[gpu]] or nvidia-container-cli: initialization error to a docker run --gpus all …, the toolkit is missing — see No CUDA device in the container. |
out of memory (cudaErrorMemoryAllocation) | Another process held VRAM while the probe ran. | nvidia-smi — see Out of memory. |
anything else — a stderr not listed here, or error reading UUID mismatch … got '<something other than None>' | Unclassified. The value after expected is a one-off challenge the validator generates per check, not a GPU UUID; an answer other than that value or None is a wrong result, not a card-mapping problem. | The generic row above: the docker run … nvidia-smi test, nvidia-smi for leftover processes, then one cycle. Include the stderr line in your ticket. |
CUDA cannot start: system not yet initialized
CUDA error 802 means the driver is loaded but the GPU's compute path is not initialised. On HGX H100/H200 boards (SXM modules behind an NVSwitch) that path is the NVLink fabric, which a separate service — NVIDIA Fabric Manager — has to bring up before any CUDA program can allocate memory. nvidia-smi does not need the fabric, so it keeps working. Three commands tell you which of three situations you are in:
nvidia-smi -q | grep -A4 -i fabric # the Fabric block: State : Completed / In Progress / Not Started / N/A
systemd-detect-virt # "none" on bare metal; kvm, vmware, … in a VM
nvidia-smi --query-gpu=name,driver_version --format=csv
| What you see | Situation | Fix |
|---|---|---|
Fabric State : In Progress or Not Started, systemd-detect-virt → none, a name without NVL or PCIe (an H100 SXM reports as NVIDIA H100 80GB HBM3, an H200 SXM as NVIDIA H200) | Bare metal with NVSwitch; Fabric Manager is not running or not installed. | systemctl enable --now nvidia-fabricmanager, then systemctl status nvidia-fabricmanager must be active (running) and the fabric state must read Completed. Fabric Manager's version has to match the driver: install the nvidia-fabricmanager-<driver major> package for your driver_version. |
systemd-detect-virt → anything but none | A VM. Who owns the NVSwitches decides: ls /dev/nvidia-nvswitch* in the guest. Present → full passthrough, the switches are yours and Fabric Manager runs in the VM. Absent → shared NVSwitch, the host's Fabric Manager owns the fabric and nothing inside the guest can start it. | Switches present: apply the bare-metal row inside the VM. Absent: ask your hosting provider to activate the fabric partition for the GPUs passed to your VM. |
systemd-detect-virt → none and Fabric Manager fails with request to query NVSwitch device information from NVSwitch driver failed, or the fabric block shows N/A, or the name is an NVL or PCIe model | No NVSwitch on this board: Fabric Manager has nothing to manage, and CUDA is failing for a reason the fabric does not explain — a broken driver or CUDA install on the host, or Fabric Manager wrongly enabled. | Stop and disable Fabric Manager (systemctl disable --now nvidia-fabricmanager). Purge and reinstall the driver release NVIDIA publishes for your card (driver_version from the command above; a version the platform cannot match is also marked invalid, see A driver version marked invalid), reboot, and re-run the docker run … nvidia-smi test from the container case. Still failing? Put dmesg | grep -i nvrm and the three command outputs in your ticket. |
Then wait one cycle. Adding an HGX node? Bring the fabric to Completed before you register it — see the quickstart requirements.
No CUDA device in the container
The host sees the GPU; the probe, which runs inside the node container, does not. Ask that container:
docker exec executor-executor-1 nvidia-smi
It must list every GPU (executor-executor-1 is the container name on the standard stack; docker ps if you renamed the compose project). Match what you see to one of these fixes:
- The container lists no GPU, or a fresh
docker run --rm --runtime=sysbox-runc --gpus all daturaai/compute-subnet-executor:latest nvidia-smianswerscould not select device driver "" with capabilities: [[gpu]]ornvidia-container-cli: initialization error— the NVIDIA Container Toolkit is missing or not registered. Reinstallnvidia-container-toolkit, runnvidia-ctk runtime configure --runtime=docker,systemctl restart docker, then recreate the node container so it starts with GPU access:cd compute-subnet/neurons/executor && docker compose up -d --force-recreate. - The container lists every GPU and the probe still saw no device — the container is missing the CUDA device node.
modprobe nvidia_uvmon the host and recreate the container the same way. stderrsays the CUDA driver version is insufficient — install a current driver release NVIDIA publishes for your card.docker runanswersunknown or invalid runtime name: sysbox-runc— the runtime itself is missing; see Sysbox setup.
Then wait one cycle.
Out of memory
Failed to allocate d_B: out of memory means another process held VRAM while the probe ran: a leftover container, a host-side job, or a rental that ended without its container stopping. nvidia-smi shows the processes and the used memory; stop whatever is not a Lium pod or default-job container and wait one cycle. If nvidia-smi on the host prints Driver/library version mismatch instead, the probe reports the same out of memory text: the driver files on disk are newer than the driver loaded in the kernel, and the fix is a reboot of the host. On a rented node, click Pause New Rentals and wait for the rental to end first. Validators report this case as its own code, GPU_VERIFY_VRAM_UNAVAILABLE (row above). When the memory belongs to a Lium pod or default job that started after the cycle's rental snapshot, validators report GPU_VERIFY_SKIPPED_RENTED instead and leave the score unchanged.
7. Rental probe: what it checks and how to fix each failed step
The probe ships with the validator but is off today. The team turns it on per validator, so a node can show RENTAL_PROBE_FAILED only after that.
The 15-minute checks prove the GPUs, the bandwidth and the ports. They do not start a customer container. A node can pass every check and still break each rental, for example when the mapped SSH port never answers. The rental probe closes that gap. The validator rents your idle node the way a customer does. While the node keeps passing, it does that at most once per interval (6 hours by default). The probe has five steps:
container_start: it starts the default rental image for your GPU and driver with the node's port mapping. The image must already be on the host. A node without it already scores0withRECOMMENDED_IMAGE_NOT_CACHEDbefore the probe runs.sshd_listen: it waits up to 90 s for SSH to answer on the mapped port.ssh_login: it logs in asrootwith a key it made for this run only.gpu_count: it runsnvidia-smi -Linside the container and counts the GPUs.teardown: it removes the container and its volume.
The probe never runs while a customer pod or a Lium Default Job is on the node. A pass removes the container and its volume.
If a step fails, the node scores 0 for that cycle and its verification is cleared, so it is off the market for that cycle. The node's detail page shows the reason code RENTAL_PROBE_FAILED. Technical details carries failed_step, the image the probe used, and the time each step took in probe_steps. Once the container exists, it also carries ssh_port, the mapped port the probe used. While the node stays idle, the validator probes it again every cycle. It is listed again as soon as a cycle passes. There is no button to press.
If every step passes but the normal removal of the container does not finish, no penalty applies. The validator removes the container by name, records RENTAL_PROBE_INCONCLUSIVE, and probes again on the next cycle.
Match failed_step to a row and run the commands on the host. <image> is the image value in Technical details.
failed_step | What happened | What to do |
|---|---|---|
container_start | The container did not start, or the create did not finish within 300 s. | Run docker run --rm --gpus all <image> nvidia-smi -L on the host. Make sure that the Docker daemon runs, that the NVIDIA container runtime is installed, and that the node's renting port range is free. |
sshd_listen | The container started but nothing listened on the mapped port within 90 s. | Port 22 inside the container must be reachable from the internet on the mapped port. ssh_port in Technical details is the port the probe used. Start a test container on that port: docker run -d --name lium-probe-test --gpus all -p <ssh_port>:22 <image>. From a machine outside your network, nc -vz <public ip> <ssh_port> must connect. If it does not, open the port on the host firewall and on the NAT. Remove the test container afterwards with docker rm -f lium-probe-test. A container left on that port breaks the next rental mapped to it. |
ssh_login | The port answered but SSH refused the validator's key. The validator writes the key into the container itself, so the usual cause is that something other than the container answered on <ssh_port>: the host's own sshd, or a NAT rule that points at another machine. | With the test container from the row above running, ssh -p <ssh_port> root@<public ip> from outside must reach the container: the login is refused, but the host key fingerprint it shows first is the container's. Compare it with docker exec lium-probe-test ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub. If it matches the host's own ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub instead, <ssh_port> lands on the host's sshd. Fix the NAT or the host firewall rule if another service answers. Remove the test container afterwards. |
gpu_count | SSH worked but nvidia-smi -L inside the container did not list every GPU the node advertises. | Run docker run --rm --gpus all <image> nvidia-smi -L on the host and compare it with nvidia-smi -L. If a card is missing, see No CUDA device in the container. |
Then wait one cycle. If the probe fails again on the same step, open a ticket as described in Before you open a ticket and include probe_steps from Technical details.
Next steps
- Grafana Dashboards — full dashboard catalog beyond Job Logs (Penalty Events, Weights, GPU Demand Analytics).
- Provider Portal — Monitoring — portal-side status indicators and embedded Grafana tab.
- Node Quickstart — full setup if you suspect a misconfiguration at the host level.