--- sidebar_position: 8.5 --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.lium.io/llms.txt > Use this file to discover all available pages before exploring further. # If your pod disappears A pod can leave **Your Pods** without you clicking DELETE, and a file on a pod's disk can stop matching the bytes you wrote to it without any error reaching your process. Both happen rarely; both are expensive when they do. This page is what to check first, and how to make sure you never lose more than the last few minutes of work. ## Why a pod can go away on its own | What you see | What happened | Where the record is | |---|---|---| | Pod gone from the list, SSH refused | **Scheduled termination** reached its time ([Scheduled termination](./scheduled-termination)) | Billing history; the termination time you set | | Pod gone, balance near zero | Your balance ran out and the platform removed the pod | Low-balance email (if enabled in Notifications); billing history | | Pod gone, nothing else changed | **Another person or agent on the same account** deleted it — every API key on an account can delete every pod on it | The events list on your account (see below) | | Pod gone, and an email about the machine a day or two earlier | **Provider reclaim** — the provider gave notice (at least 48 hours ahead) that the machine leaves the marketplace for good; when the notice period ran out the platform removed the pod permanently | The email *Your pod will be permanently removed*, sent when the provider set the notice, with the reason and the start time; billing history | | Pod still listed, status **BROKEN** | The provider force-closed the pod, or the machine went offline for good. You are credited. | [Broken pods](./broken-pods) | | Pod still listed, status **REBOOT_FAILED** | A reboot was dispatched but the container could not be recreated — typically the host was unreachable over SSH. The pod stays listed and **billed** so you can retry: reboot again once the node recovers, or delete the pod. | The status in Your Pods; the `pod-create.failed` event in the events list on your account (see below) when the validator replied — a reboot the validator never answered is marked failed after 30 minutes with no event | The third row is the one people miss. Automation that shares one account — a CI runner, a fleet script, several agents — will sooner or later remove a pod another job still relies on. Give each job its own [API key](./api-keys), name pods after their owner, and read the events list before assuming the platform lost the pod. ## Where to look 1. **Your Pods** — is the pod listed with a red status (BROKEN, REBOOT_FAILED, CREATION_FAILED), or not listed at all? 2. **Account events** — every pod create, delete, reboot and failure **performed by a key or a login on your account** is recorded with the pod's name and the time. There is no screen for this list in the dashboard yet; it is read over the API (`GET /users/me/events`, see below). A `pod-delete.success` event with a timestamp that matches the moment SSH stopped answering means a key on your account removed the pod, not the platform. The reverse does not hold: removals the platform performs on its own — balance ran out, scheduled termination, provider reclaim — are recorded without an account and **do not appear in this list**. Presence proves your own key deleted it; absence proves nothing. 3. **Billing history** — the rental row shows when billing stopped. For the platform's own removals this, together with the emails in the table above, is the record. Not released yet on the platform side (lium-platform#62; the CLI side, lium#156, shipped in lium 0.0.38): a per-pod lifecycle event that names the close reason (`user_initiated`, `scheduled_termination`, `insufficient_balance`, `executor_offline`, `executor_reclaimed`, `broken_by_provider`, `undeploy_failed`) — or `reboot_failed` with the validator's error for a pod that is still rented but failed to reboot — on the pod detail (`GET /pods/{id}`) and in the CLI (`lium describe `, `lium ps`). Until they ship, the events list is the record for deletes made through your account, and billing history plus the emails in the table above are the record for the platform's. ## Hash everything off-pod A pod is compute, not storage. Two properties follow: - The pod can be gone in a minute (any row above). Anything that exists only on the pod is gone with it. - A file on the pod's disk can silently stop matching what you wrote. Observed in the field: a video chunk uploaded to a remote store, then re-read from the pod's disk minutes later, hashed differently from the uploaded copy and decoded with visible artefacts — with **no I/O error** reported to the process. Disk health of the host in the machine's listing is not released yet (lium-platform#63, lium-io#1278); until then only a hash tells you. The habit that protects you costs one command per file: ```bash # On the pod, once per finished file, the moment it is finished — append, never rewrite: sha256sum outputs/run7.mkv >> outputs/SHA256SUMS # (re-hashing every file with `sha256sum outputs/*.mkv > SHA256SUMS` would bless a file that # has already gone bad as the new expected hash) # upload outputs/ AND the manifest to your store (Hugging Face, S3, your own server) # Anywhere, before you use a file that has been on a pod's disk for a while: sha256sum -c SHA256SUMS ``` Rules of thumb: - **Upload as you go.** Push each finished artefact off the pod the moment it exists; do not wait for the end of the job. Volumes and [Backups](./backups) cover persistent state; they are not a substitute for pushing results to a store you control. - **Hash on write, verify on read.** Record the hash when the file is created, compare against it before consuming the file again on the pod, and compare the uploaded copy against the same manifest. - **Re-fetch, do not re-use.** If a hash no longer matches, re-download the good copy from your store rather than repairing the file on the pod, and move the job to another machine. - **Keep the manifest with the data.** A `SHA256SUMS` next to the files is what lets anyone — you, a reviewer, a script — check them later.
For agents and automation: API Your account's event list. The endpoint takes the dashboard's session token as a bearer token — the `Authorization` value the web app sends on its own API calls after you log in. An API key is **not** accepted here, and there is no dashboard screen for it yet: ```bash curl https://lium.io/api/users/me/events \ -H "Authorization: Bearer " ``` Each entry carries `event_type` (`executor-rent.success`, `executor-unrent.success`, `executor-rent.failed`, `pod-reboot`, …), `event.sub_event_type` (`pod-create.success`, `pod-delete.success`, `pod-create.failed`, …), `event.pod_name`, and for failures `event.event_metadata.error`. A pod whose last event is `pod-delete.success` was removed through the API by a key on your account. Removals the platform performs itself (balance, scheduled termination, provider reclaim) are stored without an account and are not returned by this endpoint. A REBOOT_FAILED pod's cause is in the `pod-create.failed` event recorded at the time of the reboot **when the validator replied** (for example `Container creation failed due to Failed create_container (failure_step: ssh_connect)` — the validator could not reach the host). When the validator never replied, the platform moves the pod from REBOOT_PENDING to REBOOT_FAILED after 30 minutes without writing an event: a silent host leaves only the status, and no recorded cause. Verify an uploaded object against your manifest without downloading it: Hugging Face exposes the SHA-256 of every LFS file (`hf_hub` file metadata, or the `sha256` field in the repo tree listing); S3 returns the checksum you set at upload with `aws s3api head-object --checksum-mode ENABLED`.