Skip to main content

Broken pods

Sometimes a pod's status turns red and reads BROKEN. This means the GPU provider force-closed your pod — the container and your rental are already gone. Hover the status for the in-app note:

This pod was force-closed by the provider. Billing has stopped, and pods on Secure nodes are credited. You can delete it.

What it means for you​

  • Billing has stopped: a BROKEN pod is not charged. On a Secure node you've also been credited for the interrupted rental; on a Spot node nothing is credited (see When a pod is charged).
  • The pod is a dead shell. Anything that wasn't on a Volume or in a Backup is gone.

What you can do​

A broken pod can only be deleted. Every other action is disabled, because there's no live container to act on:

  • ❌ Reboot, edit, switch template, install Jupyter, add/remove SSH keys
  • ❌ Back up or restore
  • ✅ Delete — removes the pod from Your Pods
  • ✅ Rent another … node — see below

Open the pod and click DELETE.

When a pod fails to start​

When a pod fails to start because of its node (not your image), Lium first tries another node itself: the pod keeps its id and stays PENDING a little longer, its log reads "The pod did not start on its node; it is being started on another node", and it comes up on a node with the same GPU at the same or a lower price. Billing starts at that move. Only when no such node is free does the pod end CREATION_FAILED.

A pod that ends CREATION_FAILED never ran, and you are not billed for it: a pod is charged only once it reaches RUNNING (its provisioning time is charged then, not before), and the rental closes as a failed start. The full rule is the billing lifecycle table. Use Rent another node (below) to try a different machine of the same class.

A pod in FAILED is different: it started, then its SSH check failed. It is not an ended state. About 30 minutes after the failure the platform reboots the pod on its own. A FAILED pod is not charged at the 5-minute ticks, but the reboot moves it to REBOOT_PENDING, which is charged like a running pod whether or not the node answers, and that first charge counts from the pod's last charge before it failed (see the billing lifecycle table). So before you rent a second node from a FAILED pod, decide whether you want the first one back; if you keep both and the first recovers, you pay for both. Delete the FAILED pod if you do not want it back.

Before you pick the next node, three Browse Pods filters help avoid a repeat:

  • Reliability — the provider's record over recent rentals, 0–100 (90%+, 95%+ or 100% presets), counting rentals that failed to start because of the provider's nodes (a start that failed on your own image, when building, pulling or running it, or because you deleted the pod does not count); a provider with neither a billed rental nor such a failed start in the last 14 days has no score yet, and its nodes stay listed under any preset until you untick Include nodes without a reliability score yet (ticked by default).
  • Uptime — how long the node has been online on Lium (1, 7, 30 or 90+ days).
  • Download — the node's measured download speed (100, 500, 1000 or 5000+ Mbps).

Rent another node​

A pod in BROKEN, CREATION_FAILED or FAILED (a FAILED pod may still come back on its own — see above) offers Rent another <GPU> node — in the pod's ⋯ menu on Your Pods (the Errors tab) and as a button on the pod page. It opens Browse Pods filtered to the same GPU model and GPU count, with the node your pod failed on left out: the chip Not the node that failed shows above the results. Remove the chip (×) or Clear all to see that node again. The exclusion is not saved with your other filters: the GPU model and count you arrived with are kept like any filter you set, but the next time you open Browse Pods without the link the excluded node is listed again.

A CREATION_FAILED pod is removed from Your Pods three minutes after the failure, so use the link within that window; a BROKEN pod is removed automatically 24 hours after its rental closed (or as soon as you delete it).

Renting the same node again​

To rent the same machine again, delete the broken pod first. If you try to rent it while the broken pod is still there, the rental is blocked with a message that names the pod to remove — for example:

This node still has a broken pod (<pod_id>) that must be removed before you can rent it again. Please delete the broken pod first.

Delete the broken pod, then deploy as usual from Create a pod.

For agents and automation: API
# Delete a broken pod
curl -X DELETE https://lium.io/api/pods/<pod_id> \
  -H "X-API-Key: $LIUM_API_KEY"

A pod's status is BROKEN once the provider force-closes it. Any mutating call on it (reboot, edit, switch-template, install-jupyter, SSH keys, backup, restore) returns 400 — only delete succeeds. Renting the node again while your broken pod still exists also returns 400, with the broken pod's id in the message, until you delete it. Get an API key first.