--- sidebar_position: 2 --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.lium.io/llms.txt > Use this file to discover all available pages before exploring further. # Monitoring Nodes This guide covers monitoring your nodes, including status tracking and performance metrics. ## Overview Monitor your nodes to ensure optimal performance and maximize earnings: - **Status Monitoring**: Track node health and availability - **Performance Metrics**: Monitor key performance indicators - **Grafana Integration**: Access detailed logs and dashboards ## Node Status Monitoring ![Node Status Monitoring](../assets/executor-status-monitoring.png) The **Status** badge on the Nodes list and at the top of each node's detail page is computed by the portal from three inputs: the backend's ping of the node's service port (every 5 minutes, 10-second timeout), whether the validator currently lists the node as active, and the validator's most recent log line for the node (looked up over the last 2 hours). Hover the badge for the one-line reason. The same status and message come back as `computed_status` from `GET /executors` and `GET /executors/{id}` on the portal API. ### What each status means | Badge | Portal message | What it means | What to do | |---|---|---|---| | **AVAILABLE** | `N of M GPUs ready for rent` | Validated, listed, rentable. | Nothing. | | **RENTED** | `All M GPUs are currently rented` | Every GPU is under a rental. | Nothing. | | **VALIDATION PENDING** | `Waiting for first validation from validator (usually within 15 min)` | The node was added less than 30 minutes ago and no validator has reported on it yet. | Wait. Over the 175 nodes added in the 30 days to 6 Sep 2026, **Add Node → AVAILABLE took 21 minutes at the median and 64 minutes at the 90th percentile**; none was faster than 8 minutes. The wait is one validator cycle (every 15 minutes) plus the validator finishing every provider in that cycle before it publishes. A node that has never been probed at all shows this for 30 minutes, then **NOT DETECTED**, with no error line in between — the account is not connected to a coordinator; see [Troubleshooting](../troubleshooting.md#a-newly-added-node-stuck-in-validation_pending). | | **VALIDATION PENDING** | `Waiting for next validator check` | The last verdict passed but the node is not marked active yet — normal right after it recovers from a failure or a restart. | Wait one cycle (≤ 15 minutes). | | **VALIDATION PENDING** | one of `Miner not answering`, `Miner reported no nodes`, `No node accepted the validator's key`, `Validator's SSH key was not accepted`, `Validator could not reach this node`, `Node refused the validator's SSH login`, `Node failed attestation`, `Validator could not complete the check` — or `The last validation check didn't complete. The validator will retry automatically.` when the row matches none of them | The newest log line is not a verdict about this node but a problem at the miner level, and the portal reads it against your **Central Miner Server** switch. Switch **ON**: the validator talks to Lium's miner for your account — `Miner not answering` means Lium's miner did not reply within 30 s (nothing on your side); `No node accepted the validator's key` means Lium's miner asked the nodes of yours that this validator checks to accept its SSH key and none answered on its executor port within 10 s — **a fleet whose executor containers are all stopped looks to the validator like a miner with no nodes**. Switch **OFF**: `Miner not answering` means the miner your hotkey registered on-chain did not answer; `Miner reported no nodes` means your own miner answered with an empty list. The other titles: the validator's SSH key could not be installed through the miner, the node's SSH server refused the validator's login, or the check stopped before a verdict (could not connect to the node's SSH or executor port, the node failed TDX attestation, or an error on the validator's side). The red panel under the header names the cause and what to change. These miner-level rows were the largest group of error rows in the validator logs in the week to 6 Sep 2026 (78k rows across 167 provider hotkeys). | Switch **ON** with `No node accepted the validator's key`: on each node check that the executor container is running (`docker ps`), that its executor port answers from the internet and that the executor image is current; a node added in the last 30 s can be missed once. Switch **OFF** with `Miner not answering` and no miner of your own: turn the switch on at [Profile → Central Miner Server](https://provider.lium.io/settings) — Lium then runs the miner for you. Self-hosted: confirm your miner process is running and reachable from the internet and that the node is in your miner's own node list. `Node refused the validator's SSH login`: check that the executor container is the one listening on the node's SSH port. `Validator could not reach this node`: check that the node's SSH port and executor port answer from the internet. If it persists past one cycle, read the miner-level rows in [Job Logs](../troubleshooting.md#1-pull-error-details-from-grafana-job-logs). | | **VALIDATION FAILED** | the failing check, e.g. `VerifyX validation failed (network speed too slow) (last success: 3 hours ago)` | The validator ran its checks and rejected the node. It scores 0 and takes no new rentals until a clean check. | Open the node's detail page: the red **last error** panel names the check, its **Impact**, **How to fix**, and — under **Technical details** — what the validator measured. Look the reason code up in [Validator reason codes](../troubleshooting.md#6-validator-reason-codes). | | **NOT DETECTED** | `The validator has not checked this node recently. It may have been removed from the miner's node list, or the validator can no longer access it over SSH.` | No validator log for this node in the last 2 hours, and the node is older than 30 minutes. | Check that the node is still in `lium provider node list`; on the host, that `docker ps` shows the executor container and that the service port and SSH port are reachable from outside. Self-hosted: check the miner process. | | **OFFLINE** | `Node not responding to ping. Check that the machine is running and network is accessible.` | The backend's `POST /ping` on the node's service port (the `ip:port` on the row) failed at the last 5-minute check. One failed ping flips the badge; the next successful ping flips it back. A node can be OFFLINE while its Docker logs look normal — what failed is the service port answering from outside within 10 seconds, not the container. While OFFLINE the node is hidden from renters. | From a machine outside your network: `curl -m 10 -X POST http://IP:PORT/ping -H 'content-type: application/json' -d '{}'`. Any HTTP status proves the port is reachable; a timeout is the problem — check the firewall, NAT forwarding, and that the port in `neurons/executor/.env` is the one registered in the portal. | | **RECLAIMED** | `Collateral reclaim in progress. Node will be removed after completion` | The node carries a collateral-reclaim flag. Collateral is no longer required, so a healthy node showing this is stuck on a flag from an earlier reclaim, and every availability query skips it. | If you did not ask to remove the node, open a ticket through the [official support flow](../../security/official-support-and-scams.md#the-safest-way-to-contact-lium-support) with the node id; the flag is cleared by hand today. | | **PAUSING NEW RENTALS** | `New rentals are paused after the current rental ends` | You requested a pause while the node is still rented. | Nothing — see [Pausing new rentals](./managing-nodes.md#pausing-new-rentals). | | **NEW RENTALS PAUSED** | `New rentals are paused for this node` | The pause is in force: hidden from renters, no unrented incentive. | **Resume New Rentals** on the node page when you want it back. | The **Node Info** panel further down the detail page still shows the older `RENTAL_CHECK_*` flag and the `Active` / `Verified` booleans the badge is derived from; the badge supersedes them for day-to-day monitoring. ### The last error panel When the badge is **VALIDATION FAILED**, **OFFLINE**, or **VALIDATION PENDING** with a miner-level cause, the node's detail page shows a red panel under the header with: - **Happened at** — the time of the validator log line the verdict came from. - **Impact** — what the verdict does to the node (`Score set to 0`, `New rentals are disabled …`, `Unrented incentive withheld …`). - **How to fix** — the validator's remediation text for that reason code. - **Technical details** — the `what_we_saw` payload: the raw numbers behind the verdict, for example `ema_verifyx_download_speed` and `min_download_speed_mbps` for a network failure, or the container name for an orphaned rental container. - **Copy for Discord** (top right) — copies the hotkey, node id, error message, job batch id and technical details in the format the support flow asks for. ## Performance Monitoring ### Key Performance Indicators (KPIs) Track these metrics to optimize your node performance: #### 1. Uptime Metrics - **Active Time**: Time the node has been active in the system - **Availability**: Percentage of time the node is online and operational #### 2. Rental Price - **Current Rate**: Your current rental price per hour - **Market Position**: How your pricing compares to similar nodes ### Real-time emission and validator scores The [Earnings Dashboard](./earnings.md) displays settled emission per subnet epoch, including an hourly breakdown on a node's day page, and links each settled epoch to its on-chain proof. For live emission ahead of settlement, check the external [TaoMarketCap → Subnet 51](https://taomarketcap.com/subnets/51/miners) for your hotkey. For real-time validator scoring (the upstream input to emission), Lium exposes a [Grafana dashboard suite](../grafana.md) that providers can read directly. The same dashboards are embedded inside the Provider Portal on each node's detail page (the **Grafana** tab) — that's where you'll see the synthetic-job and scoring telemetry without leaving the portal. Use the [Earnings Dashboard](./earnings.md) for the resulting settled emission and the embedded Grafana for the validator-side score signals that produce it. ## Node alerts Provider notifications — an in-app inbox, Discord DMs and signed webhooks — arrive with DAH-2718 ([lium-platform#22](https://github.com/Datura-ai/lium-platform/pull/22), not released); today the portal sends no node alerts, so the Status badge above and the [Grafana dashboards](../grafana.md) are what you watch. This page will describe them when they ship. ## Reading the verification timeline Every validator cycle (about every 15 minutes, on a fixed grid) runs the same ordered list of checks on your node — upload, hardware scan, duplicate check, port checks, the VerifyX bandwidth and GPU proof, the GPU benchmark, the rental check, scoring. Today the portal shows only the verdict of the last completed cycle (the badge and the last error panel above); nothing shows which step a run is on. :::note Coming in lium-platform#120 and lium#210 The node page's **Verification** panel and `GET /executors/{id}/verification` are lium-platform#120; the CLI view, `lium provider node status ` (also `lium mine status `), is lium#210. Neither is released; nothing below works until they ship. They show where a run is: - **verifying · step 3/6 Bandwidth & GPU proof · 42 s elapsed · ~70 s left** — the validator reported the start of a run for this node and the checks should still be going. The step and the time left are **estimates**: elapsed time walked along the typical duration of each step in this node's last runs (the fleet's typical durations while the node has fewer than three of its own). The panel says which history it used. Steps before the running one are shown as done; they are inferred from time, not measured. - **checks done · waiting for the validator to publish the result** — the node's own checks should be over. The verdict appears only when the validator publishes the whole cycle, which happens after every provider in the cycle has been checked and scored — typically a few minutes after the node's own run ends. The panel shows when the last result landed relative to its cycle, and estimates this one from it. - **idle · last run passed in 2 min 40 s** — between runs. The timeline lists the steps of the last run with ✓/✗ and how long each took; only steps that took a second or more are listed, the rest are sub-second bookkeeping. A failed run shows ✗ on the step that stopped it and the reason code; steps after it never ran. The next cycle time is shown from the grid. - **not verified yet** — the validator has never reported on this node. The first check starts with the next cycle; expect `Add Node → AVAILABLE` to take about 20 minutes at the median (see [What each status means](#what-each-status-means)). The CLI view adds `--watch` (refreshing every 5 s) and `--json` with the same fields as `GET /executors/{id}/verification` (`phase`, `current`, `elapsed_s`, `eta_s`, `basis`, `steps[]`, `run`, `last_run`, `next_check_expected_at`) — route and fields are lium-platform#120, not released. ::: ## Monitoring Tools ### 1. Node Details Page Access comprehensive node information: 1. **Navigate to Node**: Go to `https://provider.lium.io/executors/{executor-id}` 2. **Review Status**: Check all status indicators 3. **Monitor Performance**: Track key metrics ### 2. Grafana Integration Access detailed monitoring dashboards: 1. **Access Grafana**: Click the **"Grafana"** tab on your node details page 2. **View Dashboards**: Access comprehensive monitoring dashboards 3. **Analyze Metrics**: Review detailed performance data ![Node Grafana Monitoring](../assets/executor-grafana-monitoring.png) :::info Grafana Access The Grafana integration provides real-time monitoring of all synthetic jobs. ::: ## Troubleshooting - A badge other than **AVAILABLE** / **RENTED**: start with [What each status means](#what-each-status-means) above, then the reason code in [Validator reason codes](../troubleshooting.md#6-validator-reason-codes). - Low rental activity on an **AVAILABLE** node: review the price against the market median shown on the node page and the [price limits](./managing-nodes.md#price-limits); a node that is often **OFFLINE** or **VALIDATION FAILED** is hidden from renters for those periods.