---
sidebar_position: 8
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lium.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

When a node is online but earning nothing — or a freshly-installed node is failing validator verification — start here. This page covers the two diagnostic surfaces providers reach for first:

1. **Grafana Job Logs** — the canonical place to read raw validator-side error messages for a specific node.
2. **The `verifyx-test` benchmark** — runs the same network, storage, and memory checks the validator runs, on demand.

If neither narrows down the problem, the [Provider Portal monitoring view](./portal/monitoring.md) and the [full Grafana suite](./grafana.md) cover the rest of the day-2 telemetry.

## A newly-added node stuck in `VALIDATION_PENDING`

If you *just* added the node and it sits at **`VALIDATION_PENDING`** — the portal may also say *"last validation check didn't complete"* — that is the pre-scoring state, not a failure. Before a node can go `AVAILABLE`, a validator has to probe it once, and that first probe only happens after your **account is connected to a coordinator**:

- **Central Provider Server (most providers).** Opt in at [Profile Settings → Central Provider](https://provider.lium.io/settings) → **Lium.io Central Provider Server** (see [Quickstart → Step 2](./quickstart.md#step-2--sign-in-to-the-provider-portal)). Until you opt in, nothing relays the validator's checks to your node, so it sits at `VALIDATION_PENDING` for 30 minutes and then flips to `NOT_DETECTED` — with no error either way, because no check has run.
- **Self-hosted.** Your own miner process must be running and connected — see [Self-hosted provider](./self-hosted-provider.md).

Once connected, validators probe every 15 minutes, and the node is listed only after the validator has finished every provider in that cycle and published: over the 175 nodes added in the 30 days to 6 Sep 2026, **Add Node → AVAILABLE took 21 minutes at the median and 64 minutes at the 90th percentile** (see [Monitoring → What each status means](./portal/monitoring.md#what-each-status-means)). During that window both `VALIDATION_PENDING` and *"last validation check didn't complete"* are expected — the probe simply hasn't run yet, so there is nothing to find in the logs below. A real verification failure only shows up in the [Job Logs](#1-pull-error-details-from-grafana-job-logs) once a probe has actually run; if the node has been `VALIDATION_PENDING` for more than an hour (past the 64-minute 90th percentile) *with* an opt-in already in place, start there.

## 1. Pull error details from Grafana Job Logs

The [**Job Logs** dashboard](https://grafana.lium.io/d/aejriu31349hcb/job-logs) is the per-node scoring log. Every validator run for your node — successful, failed, or unreachable — lands here with the exact error message the validator produced.

### Open the dashboard with your filters

1. Go to [grafana.lium.io → Job Logs](https://grafana.lium.io/d/aejriu31349hcb/job-logs).
2. In the top toolbar, set the two template variables:
   - **`miner_hotkey`** — your provider SS58 hotkey (the one registered on subnet 51).
   - **`executor_id`** — the UUID of the specific node you're debugging. You can copy this from the [Provider Portal node detail page](./portal/managing-nodes.md) or from `provider.lium.io/executors/{executor-id}`.
3. Set the time range (top-right) to cover the period you care about. Default is the last 1 hour; widen it if your node has been silent.

You can also reach the same view without leaving the portal — each node's detail page has a **Grafana** tab that pre-fills both variables for that one node (see [Monitoring Nodes → Grafana Integration](./portal/monitoring.md#2-grafana-integration)).

### Read the three log feeds

The dashboard shows three separate panels, all keyed by the same `miner_hotkey` + `executor_id`:

| Panel | When it has rows | What to look for |
|---|---|---|
| **Job Logs** | Every scoring run | The score the validator gave you, the uptime, and any non-fatal warnings. |
| **Job Error Logs (Score = 0 AND GPU > 0)** | The node was reachable but failed scoring | Synthetic-job errors, missing `sysbox-runc` runtime, GPU verification failures, network-metric failures. |
| **Machine scrape Error Logs (Score = 0 AND GPU = 0)** | The validator could not even read your machine | SSH / auth failures, unreachable host, no GPU detected, port closed. |

**Triage rule of thumb:**

- Rows in panel 3 → it is a connectivity or configuration problem (network, firewall, SSH, machine offline). Fix the host before doing anything else.
- Rows in panel 2 → the host is up but a specific check is failing. Read the error message; for verifyx network or storage failures, jump to [§2](#2-run-the-verifyx-benchmark-to-reproduce-validator-checks) and reproduce locally.
- Rows in panel 1 with non-zero scores → your node is being scored normally; revenue concerns belong in [Rewards](./rewards/index.mdx) and the [rewards FAQ](./rewards/faq.mdx).

## 2. Run the verifyx benchmark to reproduce validator checks

When the Job Logs panel 2 reports a network or storage failure, the fastest way to confirm and iterate on a fix is to run the validator's own checks locally. We ship them as a Docker image, `daturaai/verifyx-test`, that runs on the node host itself. Run it before registering a new node, and re-run it any time validator-side network or storage scoring drops.

### Quick start (network only)

```bash
docker run --gpus all --rm daturaai/verifyx-test:latest
```

Runs a 10-iteration network benchmark and prints download and upload speeds. Finishes in roughly 2–3 minutes.

### Full benchmark (network + storage + memory)

```bash
docker run --gpus all --rm daturaai/verifyx-test:latest python3 benchmark.py --full
```

Runs all three test suites. Expect 5–10 minutes due to the 5 GB storage I/O test.

### What is tested

| Test | What it measures | Minimum requirement |
|------|-----------------|---------------------|
| Network download | PyPI package download speed | **100 Mbps** as an EMA across cycles (`MIN_VERIFYX_EMA_DOWNLOAD_SPEED_MBPS` in the validator). The first cycle's sample *is* the EMA, so one cold measurement under 100 Mbps fails the first cycle; a node that then measures faster usually passes the next one. |
| Network upload | Cloudflare speedtest upload | — |
| Storage write | Sequential write throughput (5 GB) | **100 GB free space** |
| Storage read | Sequential read throughput (5 GB) | — |
| Memory allocation | Allocates 75 % of available RAM (8–128 GB range) | **8 GB RAM** |

### Interpreting results

The benchmark runs **10 iterations** and reports an Exponential Moving Average (EMA) with `alpha = 0.3` (decay = 0.7), so recent runs carry more weight. The final printed values are the EMA-smoothed numbers — a single slow run will not dominate the result.

The last section of the output also shows your hardware stats (total RAM, free disk space, utilization) from the most recent successful run.

### Prerequisites

Your host must have the **NVIDIA Container Toolkit** installed so Docker can pass GPU access through the `--gpus all` flag. If you followed the [Node setup guide](./nodes/quickstart.md), this is already configured.

```bash
# Verify the toolkit is installed
docker run --gpus all --rm nvidia/cuda:12.8.1-base-ubuntu22.04 nvidia-smi
```

## 3. Common issues

| Symptom (Job Logs message or benchmark output) | Likely cause | Fix |
|---|---|---|
| `VERIFYX_FAILED_NETWORK_SPEED_TOO_SLOW` — EMA download speed below 100 Mbps | ISP throttling or congested uplink; on a brand-new node, a single cold first sample | Re-run the benchmark above; if it is consistently under 100 Mbps, contact your hosting provider or move to a dedicated port. A first-cycle failure on a node that measures well locally clears on the next cycle. |
| Storage test fails — not enough free space | Less than 100 GB free on the node disk | Free space or resize the volume. See [Docker storage](./nodes/docker-storage.md). |
| Memory allocation fails | Less than 8 GB RAM available to Docker | Reduce other processes consuming RAM; upgrade RAM if needed. |
| `--gpus all` flag not recognised | NVIDIA Container Toolkit not installed | Follow the [Node setup guide](./nodes/quickstart.md). |
| Renters see no pod logs or GPU metrics for pods on this node; the node container's log (`docker logs executor-executor-1`) says `Refusing a signed /containers request: Signed timestamp is …s behind the executor clock; the accepted window is 300s` (or `ahead of`) | The node's clock is more than 5 minutes off. The platform signs each pod-log and pod-metrics request with the time it was made, and the node refuses one outside the window. | Sync the clock: `timedatectl set-ntp true`, then check `timedatectl` shows `System clock synchronized: yes`. The window is 5 minutes either way; fix the clock rather than widen it. |
| Job Logs panel 3 has rows; SSH / auth errors | Validator cannot reach the node | Check firewall, the node's public port, SSH service, and the credentials registered in the Provider Portal. |
| Job Logs panel 2 mentions `sysbox-runc` | Sysbox runtime missing or not the default | Install Sysbox — see [Sysbox setup](./nodes/sysbox.md). It is required; validators reject any node without it. |
| Job Logs mention `Advertised CPU cores exceed the host's physical cores` | The advertised `CPU(s)` count exceeds the kernel-present CPU population (`/sys/devices/system/cpu/present`) by more than 4 | Advertise the node's real core count. Until then, the node scores `0` in both the rented and the [unrented](./rewards/emission.mdx#unrented-pool) pool. Read both counts yourself — see [§5](#5-check-your-nodes-cpu-count). |
| Job Logs mention `gpu requires reset`, `unknown device`, `nvml error: unknown error`, or a driver/library version mismatch | The GPU fell off the bus, or the driver is half-upgraded | The node is quarantined — see [§4](#4-quarantine-your-node-was-pulled-off-the-market). |
| Job Logs mention `cannot_apply_gpu_power_cap` | The node container is not allowed to set a GPU power limit | The node stays rentable but earns no idle payout — see [GPU power cap](./nodes/gpu-power-cap.md). |
| Job Logs mention `outdated_executor_image` | The node runs an older `daturaai/compute-subnet-executor` image; the current one has been required since 2026-09-01 00:00 UTC | An idle node scores `0` and leaves the market; a rented node keeps its rental but earns no incentive — see [The standard stack, which stopped updating](./nodes/gpu-power-cap.md#the-standard-stack-which-stopped-updating). |
| Node stays `VALIDATION_PENDING` right after **Add Node**; no rows in any Job Logs panel yet | The account is not connected to a coordinator, so no validator has probed the node yet | Opt into the [Central Provider Server](https://provider.lium.io/settings) (or start your [self-hosted](./self-hosted-provider.md) miner); validators probe every 15 minutes; the listing follows 21 minutes at the median, 64 at the 90th percentile, after you connect. See [A newly-added node stuck in VALIDATION_PENDING](#a-newly-added-node-stuck-in-validation_pending). |

## 4. Quarantine: your node was pulled off the market

A fatal GPU-runtime fault does not score the node down; it **quarantines** it: the node scores `0`
and receives no new rentals until it passes a clean verification.

| Verdict | What the host reported | What fixes it |
|---|---|---|
| `GPU_RUNTIME_DEVICE_FAULT` | `gpu requires reset`, `unknown device`, or `nvml error: unknown error` | **Reset the host.** Reboot it, then check `nvidia-smi` and `dmesg` for an Xid before putting it back. A driver reconcile does not help here. |
| `GPU_RUNTIME_NVML_MISMATCH` | `driver/library version mismatch` | Reconcile the driver — the loaded kernel module and the userspace NVML library are different versions. |

**Getting back.** There is no button to press and no ticket to open: once the fault is gone, the next
clean check marks the node active again, roughly one cycle (~15 minutes) later. If it is quarantined
again straight away, the fault is still there — read the exact stderr in the
[Job Logs](#1-pull-error-details-from-grafana-job-logs) panels.

### A driver version marked invalid

A driver version the platform cannot match to a binary NVIDIA publishes is marked invalid, and every
node running it scores `0`. The mark is no longer permanent: it is re-decided about a day later, so a
version branded by mistake clears on its own — no ticket needed. If yours does not clear, the
platform still cannot verify that version; install a driver release NVIDIA publishes.

## 5. Check your node's CPU count

Read both numbers where the validator reads them — inside the node container:

```bash
docker exec executor-executor-1 lscpu | grep '^CPU(s):'
docker exec executor-executor-1 cat /sys/devices/system/cpu/present
```

`executor-executor-1` is the container name on the standard stack; run `docker ps` if you renamed
the compose project. The second command prints CPU numbers, not a count: `0-127` means 128 CPUs, and
commas mean several ranges — `0-19,40-59` is 40. An honest host matches, even one with cores
offlined.

Only an advertised count **more than 4 above** the kernel number fails. A lower count never does,
and this check alone never zeroes a node it could not measure: an unreadable `present` file, or a
scrape with no core count in it, passes.

## 6. Validator reason codes

Every verdict the validator writes carries a `reason_code`. It is the title of the red **last error** panel on the node's detail page in the Provider Portal (see [Monitoring → The last error panel](./portal/monitoring.md#the-last-error-panel)) and the `reason_code` field in the JSON after `>>>` in the Job Logs. The codes below are the ones that score a node `0` or withhold its incentive, ordered by how many distinct nodes hit each one in the week to 6 Sep 2026 (most first). Codes with severity `info` are not listed: they do not change the score.

| Reason code | What the validator saw | Effect | What to do |
|---|---|---|---|
| `GPU_VERIFY_FAILED` | The matrix-multiplication probe the validator runs inside a container on your node did not produce the expected result. | Score `0`. | `docker run --rm --runtime=sysbox-runc --gpus all daturaai/compute-subnet-executor:latest nvidia-smi` must list every GPU. Look for leftover containers or host processes holding the cards (`nvidia-smi`), then wait one cycle. |
| `SYSBOX_REQUIRED_MISSING` | The `sysbox-runc` runtime is not installed or not usable on an unrented node. | Score `0` for the cycle; repeated failures deactivate the node. | [Install Sysbox](./nodes/sysbox.md) and verify with the command on that page. |
| `VERIFYX_FAILED_NETWORK_SPEED_TOO_SLOW` | The EMA of the VerifyX download measurement is under 100 Mbps. `Technical details` shows `ema_verifyx_download_speed` and `min_download_speed_mbps`. | Score `0`. | See [§2](#2-run-the-verifyx-benchmark-to-reproduce-validator-checks). A first-cycle failure on a new node that measures well locally clears on the next cycle. |
| `SCRAPE_FAILED` / `SCRAPE_PARSE_FAILED` | The machine-spec scrape the validator runs over SSH in the executor container failed to run, or returned output it could not decrypt or parse. | Validation halted, GPU unverified, score `0`. | `docker logs executor-executor-1` for the traceback; make sure the container is up to date (`docker compose pull && docker compose up -d` in `neurons/executor`) and that the host is not out of disk or memory. |
| `INSUFFICIENT_PORTS` | Fewer than 3 of the ports in the node's renting port range are reachable from the validator. | Score `0`. | Open the renting port range you configured in `lium mine` on the firewall / NAT, or widen it in `neurons/executor/.env` and restart the container. |
| `UPLOAD_FAILED` | The validator could not copy its check scripts onto the node over SSH. | Validation halted, score `0`. | Check free disk space on the host, that the SSH port (`2200` by default) is reachable from outside, and `docker logs executor-executor-1` for SSH errors. |
| `GPU_USAGE_HIGH` / `GPU_USAGE_OUTSIDE_TENANT` / `GPU_FOREIGN_PROCESS` / `GPU_VRAM_HELD` | A GPU on an idle node is busy, holds VRAM, or runs a process that is not a Lium pod or default-job container — another miner, a host-side job, or a hidden process. | Score `0`. | Stop every non-Lium GPU workload on the host until `nvidia-smi` shows no processes and no used memory. Capacity sold on Lium must be free for the renter. |
| `ORPHANED_RENTAL_CONTAINER` | A rental ended but its container is still running on the host. | Score `0`. | `docker stop <container>` — the name is in `Technical details`. |
| `EXECUTOR_TRANSPORT_UNREACHABLE` / `FILLER_TRANSPORT_UNREACHABLE` | An SSH connection to the node dropped mid-check (network blip, connection-tracking flush). | No verdict for that cycle. | Nothing if it happens once. If it repeats, check the host's network path and the SSH port. |
| `VERIFYX_FAILED` (with a suffix: `_SSH_TRANSPORT`, `_EXECUTOR_CRASH`, `_EMPTY_RESPONSE`, `_CIPHER_REJECTED`) | The VerifyX hardware probe failed for a reason other than speed: SSH dropped, the probe process crashed on the node, returned nothing, or returned a response the validator rejected. | Score `0`. | Read `docker logs executor-executor-1` for the traceback; `dmesg` for OOM kills and `df -h` for a full disk; restart the container so it runs the current image. The validator's own debugging notes are in [`verifyx-debug.md`](https://github.com/Datura-ai/lium-io/blob/main/docs/lium-io/verifyx-debug.md). |
| `FILLER_STOPPED_BY_HOST` / `FILLER_CONTAINER_MISSING` / `FILLER_KILLED_AT_CREATE` | Something on the host stopped or removed a Lium default-job (`filler_*`) container while its run was active — repeatedly, in the `_AT_CREATE` case. | Unrented incentive withheld until a default job runs undisturbed. | Do not stop `filler_*` containers; allow-list them in any host-side cleanup tool (Watchtower, cron `docker prune`, another subnet's agent). See [Default Jobs](./portal/default-jobs.md). |
| `RENTAL_NOT_VERIFIED` | A rented pod's container could not be verified on the node. | Validation halted. | Check the pod's container is running (`docker ps`); if it is gone but the pod still shows as rented, see [Force-closing a pod](./portal/managing-nodes.md#force-closing-a-pod-mark-broken). |
| `POD_NOT_RUNNING` | A pod the backend records as running is not running on the node. | Score `0`, verification cleared. | Same as above. |
| `PROVIDER_BANNED` / `GPU_BANNED` | Your hotkey, coldkey or a GPU UUID is on the ban list. | Score `0`; running rentals finish. | See [Provider ban](./rewards/penalties.mdx#provider-ban). |
| `EXECUTOR_IMAGE_OUTDATED` | The executor container is not running the current image. | Score `0` until it is. | `cd compute-subnet/neurons/executor && docker compose pull && docker compose up -d`, or re-run `lium mine`. |
| `GPU_VERIFY_TIMEOUT` | The matrix probe did not finish in time. | Score `0`. | `nvidia-smi` for stuck processes or containers holding VRAM; reboot the host if a GPU is wedged. |
| `GPU_VERIFY_VRAM_UNAVAILABLE` *(lium-io#1335, not released)* | The matrix probe could not allocate GPU memory: another process held VRAM while it ran. Older validators report this as `GPU_VERIFY_FAILED` with "UUID mismatch … got 'None'". | Score `0`. | `nvidia-smi` for processes or leftover containers on the GPU; the next cycle retries. |
| `GPU_FAULT_PROBE_FAILED` *(lium-io#1297, not released)* | The kernel-fault probe that follows the matrix probe (random gather/scatter, atomics, a pointer chase and a copy round-trip over ~2 GB of VRAM) hit a CUDA error, returned wrong data, hung, or NVML reported new uncorrected ECC errors / remapped rows. `Technical details` names the GPU index and the error. | Warning while the probe is in shadow mode; score `0` once the validator enforces it. | `nvidia-smi -q -d ECC,ROW_REMAPPER` and the host `dmesg` for `NVRM: Xid` lines. Reset the card (`nvidia-smi -r`, with nothing running on it) and wait one cycle; a card that fails again needs reseating or replacing. |
| `GPU_UUID_CHANGED` / `SPEC_CHANGED` | The set of GPU UUIDs, or the hardware inventory, differs from the last cycle. | Verification reset, score `0` for the cycle. | Nothing if you changed the hardware on purpose — the next clean cycle re-verifies. Otherwise a card dropped off the bus: check `dmesg` and `nvidia-smi`. |
| `GPU_POWER_LIMIT_BELOW_DEFAULT` | A GPU's power limit is under 90 % of its NVML default. | Score `0`. | `nvidia-smi -pl <default>` to restore it; see [GPU power cap](./nodes/gpu-power-cap.md) for how Lium's own filler caps are told apart. |
| `GPU_DETAILS_MISMATCH` / `GPU_COUNT_ZERO` / `GPU_VRAM_MISMATCH` / `GPU_COUNT_EXCEEDS_MAX` / `GPU_MODEL_UNSUPPORTED` | The GPU count, model or VRAM the node reports is inconsistent, zero, above the per-node maximum, or not on the [supported list](./architecture.md#supported-gpus). | Score `0`. | Make the registered GPU type and count match `nvidia-smi -L`; do not modify what the executor reports. |
| `NVML_DIGEST_MISMATCH` / `NVML_DRIVER_UNKNOWN` | The NVIDIA driver library on the node does not match a known build, or the driver version is not one the platform recognises. | Score `0`; previous verification cleared. | Reinstall a driver release NVIDIA publishes; see [A driver version marked invalid](#a-driver-version-marked-invalid). |
| `GPU_RUNTIME_DEVICE_FAULT` / `GPU_RUNTIME_NVML_MISMATCH` / `GPU_WEDGED` | The GPU is not addressable, the driver is half-upgraded, or a card is pinned busy with no process. | Quarantined: no new rentals until a clean check. | [§4](#4-quarantine-your-node-was-pulled-off-the-market). |
| `CPU_QUOTA_EXCEEDS_HOST` / `CPU_TRUTH_MISMATCH` | The advertised CPU count is above what the host has. | Score `0`. | [§5](#5-check-your-nodes-cpu-count). |
| `PROVIDER_SIDE_LOAD_ABOVE_LIMIT` | A workload outside Lium containers is using the node's CPU or disk. | Score `0`. | Stop it. |
| `PORT_VERIFY_FAILED` | The validator could not reach a port it mapped on the node. | Score `0`. | Check Docker port mappings and the firewall for the renting port range. |
| `EXECUTOR_DUPLICATE` | Two registered nodes share one executor UUID. | Score `0`, verification cleared. | Deregister the duplicate in the portal. |
| `INSPECTOR_*` | The tenant-workload inspector could not run or flagged something. | Score unchanged. | Informational; if it repeats every cycle, restart the executor container so it ships the current `libinspector`. |
| `*_CONFIG_MISSING`, `MISSING_REMOTE_DIR`, `*_POLICY_MISSING` | The validator's own configuration is incomplete. | Validation halted for the cycle. | Nothing on your side. If a node shows one of these for more than two cycles, report it through the [support flow](../security/official-support-and-scams.md#the-safest-way-to-contact-lium-support). |

## Next steps

- [Grafana Dashboards](./grafana.md) — full dashboard catalogue beyond Job Logs (Penalty Events, Weights, GPU Demand Analytics).
- [Provider Portal — Monitoring](./portal/monitoring.md) — portal-side status indicators and embedded Grafana tab.
- [Node Quickstart](./nodes/quickstart.md) — full setup if you suspect a misconfiguration at the host level.