--- sidebar_position: 8.5 --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.lium.io/llms.txt > Use this file to discover all available pages before exploring further. # If your pod disappears A pod can leave **Your Pods** without you clicking DELETE, and a file on a pod's disk can stop matching the bytes you wrote to it without any error reaching your process. Both happen rarely; both are expensive when they do. This page is what to check first, and how to make sure you never lose more than the last few minutes of work. ## Why a pod can go away on its own | What you see | What happened | Where the record is | |---|---|---| | Pod gone from the list, SSH refused | **Scheduled termination** reached its time ([Scheduled termination](./scheduled-termination)) | Billing history; the termination time you set | | Pod gone, balance near zero | Your balance ran out and the platform removed the pod | Low-balance email (if enabled in Notifications); billing history | | Pod gone, nothing else changed | **Another person or agent on the same account** deleted it — every API key on an account can delete every pod on it | The events list on your account (see below) | | Pod gone, and an email about the machine a day or two earlier | **Provider reclaim** — the provider gave notice (at least 48 hours ahead) that the machine leaves the marketplace for good; when the notice period ran out the platform removed the pod permanently ([Provider maintenance on your pod](./provider-maintenance)) | The email *Your Lium pod `` will be removed on ``*, sent when the provider set the notice, with the reason and the removal time; billing history | | Pod still listed, status **BROKEN** | The provider force-closed the pod, or the machine went offline for good. You are credited. | [Broken pods](./broken-pods) | | Pod listed as **CREATION_FAILED**, or gone about three minutes after it was **PENDING** | The rent failed: the node could not build or start the container. The platform removes a pod that did not start about three minutes after the failure. | The pod page: a red panel names the failed step in plain words (for example *Your Dockerfile did not build*), shows the node's error and, for a custom Dockerfile, the last lines of the build with a copy button. The panel stays readable after the pod is removed. | | Pod still listed, status **REBOOT_FAILED** | A reboot was dispatched but the container could not be recreated — typically the host was unreachable over SSH. The pod stays listed and **billed** so you can retry: reboot again once the node recovers, or delete the pod. It stays billed whatever the error says, because the node stays reserved for you. Billing stops only while the node is marked inactive (the validator found it broken, or has not reported it for an hour), and starts again when the node is active again; you are not charged for that time later. See [When a pod is charged](./billing#when-a-pod-is-charged). | The status in Your Pods; the `pod-create.failed` event in the events list on your account (see below) when Lium got a reply — a reboot that got no reply is marked failed after 30 minutes with no event | The third row is the one people miss. Automation that shares one account — a CI runner, a fleet script, several agents — will sooner or later remove a pod another job still relies on. Give each job its own [API key](./api-keys), name pods after their owner, and read the events list before assuming the platform lost the pod. ## Where to look 1. **Your Pods** — is the pod listed with a red status (BROKEN, REBOOT_FAILED, CREATION_FAILED), or not listed at all? 2. **Account events** — every pod create, delete, reboot and failure **performed by a key or a login on your account** is recorded with the pod's name and the time. There is no screen for this list in the dashboard yet; it is read over the API (`GET /users/me/events`, see below). A `pod-delete.success` event with a timestamp that matches the moment SSH stopped answering means a key on your account removed the pod, not the platform. The reverse does not hold: removals the platform performs on its own — balance ran out, scheduled termination, provider reclaim — are recorded without an account and **do not appear in this list**. Presence proves your own key deleted it; absence proves nothing. 3. **Billing history** — the rental row shows when billing stopped. For the platform's own removals this, together with the emails in the table above, is the record. When a pod is removed, its row is deleted with it, so `GET /pods/{id}` answers 404 from then on. The close reason is in the pod's event log, `GET /pods/{id}/events`, which still answers for a removed pod: the `pod-lifecycle` entry with `to_status` `DELETED` carries the `reason` (`user_initiated`, `scheduled_termination`, `insufficient_balance`, `executor_offline`, `executor_reclaimed`, `broken_by_provider`, `undeploy_failed`). A pod that is still rented but failed to reboot keeps its row: its `reboot_failed` entry with the error is in the same event log and is the `last_event` on `GET /pods/{id}`. The events list on your account is the record for deletes made through your account, and billing history plus the emails in the table above are the record for the platform's. ## Hash everything off-pod A pod is compute, not storage. Two properties follow: - The pod can be gone in a minute (any row above). Anything that exists only on the pod is gone with it. - A file on the pod's disk can silently stop matching what you wrote. Observed in the field: a video chunk uploaded to a remote store, then re-read from the pod's disk minutes later, hashed differently from the uploaded copy and decoded with visible artefacts — with **no I/O error** reported to the process. The machine's listing carries the host's disk health (`disk_health_ok` and `specs.disk_health` on `GET /executors`). `disk_health_ok` is false only when the Docker disk is read-only or refused a write; it can still be true when the write probe was skipped. It does not see silent corruption. For your own bytes, only a hash tells you. The habit that protects you costs one command per file: ```bash # On the pod, once per finished file, the moment it is finished — append, never rewrite: sha256sum outputs/run7.mkv >> outputs/SHA256SUMS # (re-hashing every file with `sha256sum outputs/*.mkv > SHA256SUMS` would bless a file that # has already gone bad as the new expected hash) # upload outputs/ AND the manifest to your store (Hugging Face, S3, your own server) # Anywhere, before you use a file that has been on a pod's disk for a while: sha256sum -c SHA256SUMS ``` Rules of thumb: - **Upload as you go.** Push each finished artefact off the pod the moment it exists; do not wait for the end of the job. Volumes and [Backups](./backups) cover persistent state; they are not a substitute for pushing results to a store you control. - **Hash on write, verify on read.** Record the hash when the file is created, compare against it before consuming the file again on the pod, and compare the uploaded copy against the same manifest. - **Re-fetch, do not re-use.** If a hash does not match, re-download the good copy from your store and move the job to another machine. - **Keep the manifest with the data.** A `SHA256SUMS` next to the files is what lets anyone — you, a reviewer, a script — check them later.
For agents and automation: API Your account's event list. After sign-in the browser sends HttpOnly cookies, not an `Authorization` header. Read the same list with an API key (`X-API-Key`), as in [API keys](./api-keys.md): ```bash curl -s https://lium.io/api/users/me/events -H "X-API-Key: $LIUM_API_KEY" \ | jq -r '.[] | "\(.created_at) \(.event_type) \(.event)"' ``` Each entry carries `event_type` (`executor-rent.success`, `executor-unrent.success`, `executor-rent.failed`, `pod-reboot`, …), `event.sub_event_type` (`pod-create.success`, `pod-delete.success`, `pod-create.failed`, …), `event.pod_name`, and for failures `event.event_metadata.error`. A pod whose last event is `pod-delete.success` was removed through the API by a key on your account. Removals the platform performs itself (balance, scheduled termination, provider reclaim) are stored without an account and are not returned by this endpoint. A REBOOT_FAILED pod's cause is in the `pod-create.failed` event recorded at the time of the reboot **when Lium got a reply** (for example `Container creation failed due to Failed create_container (failure_step: ssh_connect)` — Lium could not reach the host). When no reply came, the platform moves the pod from REBOOT_PENDING to REBOOT_FAILED after 30 minutes without writing an event: a silent host leaves only the status, and no recorded cause. Verify an uploaded object against your manifest without downloading it: Hugging Face exposes the SHA-256 of every LFS file (`hf_hub` file metadata, or the `sha256` field in the repo tree listing); S3 returns the checksum you set at upload with `aws s3api head-object --checksum-mode ENABLED`.