If your pod disappears

A pod can leave Your Pods without you clicking DELETE, and a file on a pod's disk can stop matching the bytes you wrote to it without any error reaching your process. Both happen rarely; both are expensive when they do. This page is what to check first, and how to make sure you never lose more than the last few minutes of work.

Why a pod can go away on its own

What you see	What happened	Where the record is
Pod gone from the list, SSH refused	Scheduled termination reached its time (Scheduled termination)	Billing history; the termination time you set
Pod gone, balance near zero	Your balance ran out and the platform removed the pod	Low-balance email (if enabled in Notifications); billing history
Pod gone, nothing else changed	Another person or agent on the same account deleted it — every API key on an account can delete every pod on it	The events list on your account (see below)
Pod gone, and an email about the machine a day or two earlier	Provider reclaim — the provider gave notice (at least 48 hours ahead) that the machine leaves the marketplace for good; when the notice period ran out the platform removed the pod permanently (Provider maintenance on your pod)	The email Your Lium pod <name> will be removed on <date>, sent when the provider set the notice, with the reason and the removal time; billing history
Pod still listed, status BROKEN	The provider force-closed the pod, or the machine went offline for good. You are credited.	Broken pods
Pod still listed, status REBOOT_FAILED	A reboot was dispatched but the container could not be recreated — typically the host was unreachable over SSH. The pod stays listed and billed so you can retry: reboot again once the node recovers, or delete the pod.	The status in Your Pods; the pod-create.failed event in the events list on your account (see below) when the validator replied — a reboot the validator never answered is marked failed after 30 minutes with no event

The third row is the one people miss. Automation that shares one account — a CI runner, a fleet script, several agents — will sooner or later remove a pod another job still relies on. Give each job its own API key, name pods after their owner, and read the events list before assuming the platform lost the pod.

Where to look

1. Your Pods — is the pod listed with a red status (BROKEN, REBOOT_FAILED, CREATION_FAILED), or not listed at all?
2. Account events — every pod create, delete, reboot and failure performed by a key or a login on your account is recorded with the pod's name and the time. There is no screen for this list in the dashboard yet; it is read over the API (GET /users/me/events, see below). A pod-delete.success event with a timestamp that matches the moment SSH stopped answering means a key on your account removed the pod, not the platform. The reverse does not hold: removals the platform performs on its own — balance ran out, scheduled termination, provider reclaim — are recorded without an account and do not appear in this list. Presence proves your own key deleted it; absence proves nothing.
3. Billing history — the rental row shows when billing stopped. For the platform's own removals this, together with the emails in the table above, is the record.

Not released yet on the platform side (lium-platform#62; the CLI side, lium#156, shipped in lium 0.0.38): a per-pod lifecycle event that names the close reason (user_initiated, scheduled_termination, insufficient_balance, executor_offline, executor_reclaimed, broken_by_provider, undeploy_failed) — or reboot_failed with the validator's error for a pod that is still rented but failed to reboot — on the pod detail (GET /pods/{id}) and in the CLI (lium describe <pod>, lium ps). Until they ship, the events list is the record for deletes made through your account, and billing history plus the emails in the table above are the record for the platform's.

Hash everything off-pod

A pod is compute, not storage. Two properties follow:

• The pod can be gone in a minute (any row above). Anything that exists only on the pod is gone with it.
• A file on the pod's disk can silently stop matching what you wrote. Observed in the field: a video chunk uploaded to a remote store, then re-read from the pod's disk minutes later, hashed differently from the uploaded copy and decoded with visible artefacts — with no I/O error reported to the process. Disk health of the host in the machine's listing is not released yet (lium-platform#63, lium-io#1278); until then only a hash tells you.

The habit that protects you costs one command per file:

# On the pod, once per finished file, the moment it is finished — append, never rewrite:
sha256sum outputs/run7.mkv >> outputs/SHA256SUMS
# (re-hashing every file with `sha256sum outputs/*.mkv > SHA256SUMS` would bless a file that
#  has already gone bad as the new expected hash)
# upload outputs/ AND the manifest to your store (Hugging Face, S3, your own server)

# Anywhere, before you use a file that has been on a pod's disk for a while:
sha256sum -c SHA256SUMS

Rules of thumb:

• Upload as you go. Push each finished artefact off the pod the moment it exists; do not wait for the end of the job. Volumes and Backups cover persistent state; they are not a substitute for pushing results to a store you control.
• Hash on write, verify on read. Record the hash when the file is created, compare against it before consuming the file again on the pod, and compare the uploaded copy against the same manifest.
• Re-fetch, do not re-use. If a hash no longer matches, re-download the good copy from your store rather than repairing the file on the pod, and move the job to another machine.
• Keep the manifest with the data. A SHA256SUMS next to the files is what lets anyone — you, a reviewer, a script — check them later.

For agents and automation: API

Your account's event list. The endpoint takes the dashboard's session token as a bearer token — the Authorization value the web app sends on its own API calls after you log in. An API key is not accepted here, and there is no dashboard screen for it yet:

curl https://lium.io/api/users/me/events \
  -H "Authorization: Bearer <your dashboard session token>"

Each entry carries event_type (executor-rent.success, executor-unrent.success, executor-rent.failed, pod-reboot, …), event.sub_event_type (pod-create.success, pod-delete.success, pod-create.failed, …), event.pod_name, and for failures event.event_metadata.error. A pod whose last event is pod-delete.success was removed through the API by a key on your account. Removals the platform performs itself (balance, scheduled termination, provider reclaim) are stored without an account and are not returned by this endpoint.

A REBOOT_FAILED pod's cause is in the pod-create.failed event recorded at the time of the reboot when the validator replied (for example Container creation failed due to Failed create_container (failure_step: ssh_connect) — the validator could not reach the host). When the validator never replied, the platform moves the pod from REBOOT_PENDING to REBOOT_FAILED after 30 minutes without writing an event: a silent host leaves only the status, and no recorded cause.

Verify an uploaded object against your manifest without downloading it: Hugging Face exposes the SHA-256 of every LFS file (hf_hub file metadata, or the sha256 field in the repo tree listing); S3 returns the checksum you set at upload with aws s3api head-object --checksum-mode ENABLED.