Skip to main content

If your pod disappears

A pod can leave Your Pods without you deleting it. Here is how to find out why, and how to never lose more than a few minutes of work.

Why a pod goes away​

What you seeWhat happened
Pod gone at a time you setScheduled termination
Pod gone, balance near zeroYour balance ran out
Pod gone, nothing else changedSomeone or an agent on your account deleted it: any API key can delete any pod
Pod gone after an e-mail about the machineThe provider took the machine off after notice (Provider maintenance)
Status BROKENThe provider force-closed it (Broken pods)
Status CREATION_FAILEDThe pod could not start. The pod page shows which step failed.
Status followed by · NODE OFFLINEThe node stopped answering. Once the validator finds it down or it is marked inactive, the badge adds billing paused and the pod is not charged until the node is back. The pod page shows since when it is offline and how often it came back in the last 30 days, to help you choose between waiting and renting another node.
Status REBOOT_FAILEDA reboot could not recreate the container. The pod stays reserved; billing may pause when a check identifies its node as unreachable. Reboot again or delete it.

Where to look​

Your Pods

Is the pod listed with a red status, or gone?

Account events

Run the command below. A pod-delete.success at the time the pod went away means a key on your account deleted it. Removals by the platform don't show here.

curl -s https://lium.io/api/users/me/events -H "X-API-Key: $LIUM_API_KEY" \
| jq -r '.[] | "\(.created_at) \(.event_type) \(.event)"'

Billing history

The rental row shows when billing stopped.

Give each job or agent its own API key so you can tell them apart.

Keep your work safe​

A pod is compute, not storage.

  • Upload results as you go to a store you control (S3, Hugging Face, your server).
  • Hash each file when it's finished and check the hash before reusing it:
sha256sum outputs/run7.mkv >> outputs/SHA256SUMS # once per finished file
sha256sum -c SHA256SUMS # before you use the files again
  • If a hash doesn't match, download the good copy again and move to another machine.