Penalties
When a rental fails because of a problem on your node, Lium withholds that rental's fees for the day it fails and the day before, and refunds the renter the same amount. This page describes what triggers a penalty, what exactly is withheld, and what to do if you believe a penalty is wrong.
Penalties apply to rental fees only. Subnet emission — the alpha stake that accrues on your hotkey each tempo — is never touched by a penalty.
What a penalty withholds
A penalty applies to the single rental that failed, not to your whole fleet, and it covers a window of 2 UTC days: the day the penalty is applied (day D) and the day before it (D-1).
| What | Effect |
|---|---|
| Rental fees for that rental, days D and D-1 | Withheld — not paid out to you |
| The renter | Refunded the same amount |
| Payouts already settled (D-2 and older) | Untouched — never clawed back |
| Subnet emission on your hotkey | Untouched |
| Your other rentals and other nodes | Untouched |
Payouts are settled with a delay of 2 days (see Payouts). That delay is what makes this window possible: when a penalty is applied, the affected days have not been paid out yet, so no money has to be taken back.
A long rental can be penalized more than once, once per outage. Each penalty covers the same two-day window (D and D-1) ending on the day of its own trigger; fees an earlier penalty already covered are never withheld twice, so a second trigger for the same outage withholds only fees billed since the earlier penalty.
What triggers a penalty
Three events apply a penalty automatically:
| Reason code | What happened |
|---|---|
POD_UNDEPLOY_FAILED | The validator could not undeploy or clean up the pod on your node and ran out of retries, while the validator had verified your node within the last 30 minutes (or verified it during the retries). The node was reachable and the container was still not removed. |
EXECUTOR_INACTIVE_MID_RENTAL | The validator has not verified your node for more than 1 hour while a rental was active (the staleness check, which runs every 15 minutes). Also applied, through the same rule and the same confirmation, when the validator gives up removing a pod and your node has not been verified for 30 minutes or more, or has no verification on record. |
BROKEN_BY_PROVIDER | You marked a stuck rented pod as broken in the Provider Portal, which force-closes the rental. |
A fourth code, MANUAL, marks a penalty applied by the Lium team by hand. It is not an automatic
trigger.
A fifth code, PROVIDER_KILL, records a force-close that had nothing to withhold: you marked a pod
broken while it was starting or running on your listed node, before any of its rental was billed.
It withholds nothing and refunds nothing (amount_withheld = 0), so it does not change your penalty
coverage or tier. On a Secure rental it counts as a penalized rental:
- It lowers your public reliability score (the score renters filter the listing by, and one of the tie-breaks when the API picks a node for a rental).
- Until it is reverted, or for 30 days, it is an open penalty on the node, and the Lium team's removal of a stale node refuses that node.
On a Spot-tier rental the event is a dry run: it blocks no removal, and it counts in the score only when the rental was forced Spot (see the note under Effect on your node tier).
With the quick-close rule on, closing a running rental on a Secure node applies
that rule's Spot period and writes no PROVIDER_KILL event (see
Force-closing a pod). A pod that was
still starting always gets the PROVIDER_KILL event.
Force-closing a genuinely stuck pod is still the right action — it frees your node and ends a rental the renter cannot use. The penalty is what refunds the renter for that failed rental.
Two windows, one reason code
EXECUTOR_INACTIVE_MID_RENTAL has two thresholds. The staleness check (every 15 minutes) applies it after 60 minutes
without a verified check. When the validator gives up removing a pod, the same code applies after
30 minutes without a verified check (or when the node has no verification on record). The delete
path is shorter because it has a second signal: the node also failed to answer the delete. The text
of the validator's reply plays no part in the decision — a reply that timed out, an empty reply and a
reply that reads "no executor accepted the SSH key" (older validators wrote "Invalid executor id") are
treated the same; only the node's verification state counts. The reply text is kept on the pod's event record. Under both thresholds the penalty runs
through the same rule below (the check on the failed reports) and the same
two-hour confirmation. Only the staleness check marks the node inactive; the give-up path decides the
penalty and leaves the listing to that check.
EXECUTOR_INACTIVE_MID_RENTAL is decided from what the validator reported about your node since it
last verified it. The node is still marked inactive, but the penalty is not applied when every
failed check in that window stopped before the validator looked at the rented pod — its
upload of the validation files failed (UPLOAD_FAILED) or its machine-spec scrape did not return
the specs (SCRAPE_FAILED, which newer validators report as SCRAPE_FAILED_NO_GPU,
SCRAPE_FAILED_DRIVER, SCRAPE_FAILED_ON_HOST, SCRAPE_TIMEOUT or SCRAPE_TRANSPORT_FAILED; all
six count the same way). Any other reason the validator recorded (a pod not running, a GPU or
spec change, an SSH session it could not complete), no report at all, or a report without a reason
keeps the penalty. The reasons the decision was made from are stored with the penalty, so the team
can check them when you dispute it.
A penalty is confirmed before it is charged
An automatic EXECUTOR_INACTIVE_MID_RENTAL is a proposal first (the other reason codes are
applied as before). The node is marked inactive at once, but the penalty itself waits two hours (a platform setting the Lium team can change) and is
confirmed only then:
- If the validator verified your node again in the meantime — a passed check of any kind — the proposal is voided. Nothing is withheld, nothing is refunded, and no penalty event is written; the decision is kept in the platform logs with the time of the passing check.
- If the node still has no passing check, the penalty is applied for the rentals that were open when the node went inactive, over the same two days it would have covered at that moment. A renter closing the pod during the wait does not cancel it.
- While the Lium team has the penalty pipeline on incident hold (a validator restart, a platform fault that makes many nodes look inactive at once), confirmations wait for the hold to end and are then re-checked the same way. The hold is set automatically when, within 30 minutes (the check's current and previous 15-minute runs), the staleness check flips at least 1% of the fleet across at least 3 providers (a run that flips a single node is not counted), because that pattern has been a platform fault before (a backend bug on 26 June 2026 penalised 26 nodes of 14 providers in one check; 25 were reverted).
What a penalty event records
Every applied EXECUTOR_INACTIVE_MID_RENTAL penalty stores its evidence next to the amount, so a
dispute is a read, not a guess. Other reason codes record the billed days, the affected rows and the
trigger's own context (for an undeploy failure, the pod, the retry count and the time of the node's
last passing check), not the fields below.
Field in details | Meaning |
|---|---|
trigger | Which path raised it: staleness_sweep (no verified check for over an hour), undeploy_give_up (a pod removal ran out of retries on a node not verified for 30 minutes) or validator_reset:<reason> (a validator check cleared the node, e.g. POD_NOT_RUNNING). |
validator_reports | For a staleness_sweep or undeploy_give_up trigger: the reason codes and count of every failed check the validator sent about the node between its last passing check and the decision, and the newest 48 of them with timestamps. A validator reset has only the confirm-time window below. |
evidence.decided_at / evidence.confirmed_at | When the node was called inactive and when the penalty was confirmed. |
evidence.validator_reports_at_confirm | The same window read again at confirmation time. |
evidence.validator | For a validator reset: what the validator sent about the check that fired, when it sends it — a validator with the reporting change (lium-io 1355) names the check (reason_code, check_id) and for POD_NOT_RUNNING adds what it could read from the container: its state, exit code and whether it was OOM-killed, or — when docker inspect failed on the host — only diagnostics_capture_error (Docker's message) and container_missing (true when Docker says the container is gone), no exit code and no OOM flag; null from a validator without it. |
evidence.sweep | For a staleness_sweep trigger: how many nodes and providers the same staleness check run flipped, how many nodes and providers its runs flipped in the last 30 minutes, the fleet size and node threshold (1% of the fleet) it was measured against, and whether that set an incident hold. |
Penalties recorded as a dry run
Some events are logged with dry_run = true and no money is withheld:
- Spot-tier rentals. A rental that started while the node was Spot is never charged a penalty. The tier is frozen when the rental starts, so a rental that started on a Secure node can still be penalized even if the node becomes Spot later.
- Scheduled notice periods. If you set a notice period at least 24 hours ahead and then reclaim the node, the undeploy-failure and inactivity penalties are waived. The waiver extends a short time past the end of the period, because these events are detected with a delay.
Two further cases skip the penalty entirely, with nothing recorded at all: the rental produced no
billable usage, or the affected days have already been paid out. The exception is a pod you
force-close before anything was billed: that is recorded as PROVIDER_KILL (above), as a dry run
when the rental was Spot-tier.
A validator POD_NOT_RUNNING reset counts against your node only when the pod it names is RUNNING.
Any other status means the platform is operating on the pod (for example REBOOT_PENDING or
REBOOT_FAILED: a reboot, an edit or a template switch is in progress or has failed), so the reset is
not held against the node: the node stays listed and no penalty is proposed. The backend logs
pod_not_running_platform_operation with the pod and its status. This also covers a reboot that the
validator never answers, for example because the node is down: after 30 minutes the pod is marked
REBOOT_FAILED, stays billed, and its resets are skipped. Such a node is still caught by the hourly
inactivity check: an hour with no successful validation delists it, stops billing and proposes the
EXECUTOR_INACTIVE_MID_RENTAL penalty. A reset that names no pod of your node (from an older
validator, or naming a pod the platform has deleted) counts against the node. This check is in effect.
Where to see your penalties
The Penalty Events dashboard
lists every penalty; narrow it down with the miner_hotkey and executor_id variables at the top
of the dashboard. The columns that matter:
| Column | Meaning |
|---|---|
reason_code | Which event caused the penalty |
amount_withheld | How much was withheld for that rental |
dry_run | true = recorded only, false = actually charged |
action | apply, or revert if the penalty was undone |
See Grafana Dashboards for the rest of the dashboard catalog.
Effect on your node tier
Penalties also feed the reliability rule that decides your tier. Penalty coverage is the total amount withheld by penalties over the trailing 14 days, as a share of your billed rental revenue in the same window. If it rises above 20%, all of your nodes move from Secure to Spot and stop earning subnet incentive until your coverage drops back below 20%. It is calculated across all of your nodes together. Every penalty that has not been reverted counts, so a rental penalized for two outages adds both amounts; reverting one of them removes that amount only. See Node Tier for the full rule.
Penalties recorded with dry_run = true withhold no money, but the two dry-run cases weigh differently
on penalty coverage. A dry-run on a rental the platform contracted as forced Spot (force_spot, for
example after a demotion) keeps its recorded amount and counts toward coverage, so a demoted provider stays
demoted while penalties continue; a rental on a node you set to Spot yourself is left out of coverage by
tier. A penalty waived for a scheduled notice period is recorded with amount_withheld = 0 and does not
count toward coverage; the amount it would have withheld is kept in the event's details.amount_waived.
When both apply, a forced-Spot rental hit during a notice period, the tier rule wins and the amount counts.
Provider ban
A ban is not a penalty: the Lium team applies it by hand, for network abuse, matched by hotkey, coldkey or GPU UUID — so it follows the provider across all their nodes.
While a ban is in force, matching nodes earn no subnet incentive and take no new rentals, and those
with nothing running are deactivated. A rental already running is never torn down: it finishes, and
its rental fees are paid out to you day by day while it runs. The final two billing days of a rental
of a banned provider are withheld once it ends: the Earnings page shows them declined with the
reason provider_banned. Earlier days are paid as usual. A day already withheld stays withheld
after the ban is lifted; a day whose payout runs after the lift is paid.
To find out which incident caused a ban, or to dispute it, use the same support flow as for a disputed penalty, below.
If you think a penalty is wrong
A penalty can be reverted. When it is, the withheld amount is paid to you and the penalty is removed from your penalty coverage. A rental with more than one penalty is reverted one penalty at a time, the newest one first. If the penalized day was paid out in the meantime, the amount is added to a later payout instead. A final billing day withheld under a provider ban stays withheld when a penalty on it is reverted: a day the payout had already withheld before the penalty, whatever the ban's state at the revert, and a day penalized before its payout while the ban is still in force at the revert. Open a support ticket through the official Discord support flow with:
- the node id (or hotkey) and the pod id,
- the time the penalty was applied, from the dashboard,
- why you believe it is wrong — for example the node was reachable the whole time, or the pod had already been removed.
The team reads the evidence stored on the event (see What a penalty event records above) and reverts the penalty if it was applied in error.
Reducing penalties
- Keep the node verifiable: most
EXECUTOR_INACTIVE_MID_RENTALevents come from the agent or the host going silent or unreachable over SSH, not from a GPU problem. See Troubleshooting. - Schedule a notice period before planned maintenance on a rented node.
- Switch nodes you cannot keep stable to Spot. They earn idle pay only while a Lium Default Job runs on them, but an interrupted rental on them costs you nothing.