---
sidebar_position: 4
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lium.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Sysbox

[Sysbox](https://github.com/nestybox/sysbox) is a container runtime that lets customers run Docker-in-Docker securely inside pods — without `--privileged` mode. Many workloads (custom image builds, CI/CD pipelines, system-level tooling) require it.

:::danger Sysbox is required
Every validator probes every node for the `sysbox-runc` runtime. Nodes missing it are rejected — they earn **no emission and cannot be rented**. Sysbox is a hard requirement for every Lium node, not an optional optimization. Setup takes ~5 minutes.
:::

## Requirements

| Component | Requirement |
|---|---|
| Kernel | **5.19 or newer** (6.x recommended). Older kernels cannot pass GPUs into a sysbox container — see below. |
| Docker | 27.x or 28.x works out of the box. 29.x needs daemon settings — see [Docker 29](#docker-29-two-required-settings). |
| Docker storage driver | `overlay2` — check with `docker info --format '{{.Driver}}'`. On ZFS, and on some btrfs setups, the filesystem has no ID-map support and GPU passthrough breaks the same way an old kernel breaks it. |
| Architecture | x86_64 |

### Kernel 5.19+ is mandatory for GPUs

Sysbox needs **ID-mapped mounts**, which the kernel supports for overlayfs only from 5.19 onward. On an older kernel — Ubuntu 22.04 ships 5.15 by default — sysbox falls back to shiftfs and runs the container rootfs from a different path (`/var/lib/sysbox/shiftfs/<uuid>`). The NVIDIA hook doesn't know about that move and still targets the original overlay2 path, which is not there in the container's mount namespace:

```
nvidia-container-cli: mount error: /var/lib/docker/overlay2/<id>/merged/proc/driver/nvidia: no such file or directory
```

Check your kernel:

```bash
uname -r
```

If it is below `5.19`, install the HWE kernel and reboot:

```bash
sudo apt-get update
sudo apt-get install -y linux-generic-hwe-22.04
sudo reboot
```

Before you reboot:

- **Stop any rentals.** Rebooting a node under an active rental costs you a penalty.
- **Check that the NVIDIA driver is DKMS-managed** — `dkms status` should list an `nvidia` module. A driver installed from a `.run` file without DKMS will not load on the new kernel, and the node comes back with no GPUs at all. Reinstall the driver after the reboot if that happens.
- The old kernel stays in the GRUB menu, so you can boot back into it if something goes wrong.
- **Ubuntu 20.04 has no upgrade path here** — its HWE kernel is 5.15. You need 22.04 or newer.

After the reboot, confirm both `uname -r` and `nvidia-smi` before re-running the installer.

In our tests, upgrading the NVIDIA driver, the Container Toolkit, or Docker on its own did not fix this — only the kernel did.

## Install

The `lium-io` repo ships an installer that takes care of NVIDIA Container Toolkit + Sysbox in one go. From any Ubuntu host:

```bash
curl -fsSL https://raw.githubusercontent.com/Datura-ai/lium-io/main/neurons/executor/nvidia_docker_sysbox_setup.sh | sudo bash
```

Or, if you already have the [`lium-io` repo](https://github.com/Datura-ai/lium-io) cloned locally:

```bash
cd lium-io/neurons/executor
chmod +x nvidia_docker_sysbox_setup.sh
sudo ./nvidia_docker_sysbox_setup.sh
```

Confirm `/etc/docker/daemon.json` includes the sysbox runtime:

```json
{
  "runtimes": {
    "sysbox-runc": {
      "path": "/usr/bin/sysbox-runc"
    }
  }
}
```

Restart Docker:

```bash
sudo systemctl restart docker
```

## Verify

Run the same command our validator uses:

```bash
sudo docker run --rm --runtime=sysbox-runc --gpus all daturaai/compute-subnet-executor:latest nvidia-smi
```

If you see `nvidia-smi` output — you're good.

:::note `docker` and `sudo docker` can reach different daemons
The installer above runs under `sudo`, so verify under `sudo` too. Rootless Docker, or a `DOCKER_HOST` set in your shell, points your plain `docker` client at another daemon. That daemon has no `sysbox-runc` and answers `unknown or invalid runtime name: sysbox-runc`, which looks like a failed install. Compare `docker context ls` with `sudo docker context ls` — contexts are per-user, so the two clients can differ.
:::

## Docker 29: two required settings

| Docker | Status |
|---|---|
| 27.x, 28.x | Works as is. What we run and test against. |
| 29.0–29.1 | Untested. |
| 29.2.x–29.4.x | Works with `cdi` off. |
| 29.5.0 and later | Works with `cdi` **and** `time-namespaces` off — confirmed by a provider on 29.7.0. |

Docker 29 broke sysbox twice, in two unrelated ways. On **29.5.0 and later**, turn both off with one block in `/etc/docker/daemon.json`:

```json
{
  "features": {
    "cdi": false,
    "time-namespaces": false
  }
}
```

On **29.2.x–29.4.x**, set only `"cdi": false` — the `time-namespaces` key appeared in 29.5 and is untested on older daemons.

**Stop any rentals first.** Restarting the daemon kills every running container on the host, and a pod killed under an active rental costs you a penalty. Then restart Docker and re-run the verify command:

```bash
sudo systemctl restart docker
```

Both settings are reversible — remove the `features` block and restart Docker to go back. `cdi: false` is daemon-wide, though: anything else on the host that gets its devices through CDI stops getting them that way. On a Lium node that is fine, since the NVIDIA legacy hook does the injection.

### CDI: Docker 29.2.0 and later

Sysbox + GPU fails with a `permission denied` error:

```
OCI runtime create failed: ... failed to open OCI spec file: ... permission denied
```

Docker 29.2 [enables CDI (Container Device Interface) by default](https://docs.docker.com/reference/cli/dockerd/#disable-cdi-devices), routing `--gpus` through CDI — incompatible with sysbox's user namespace.

### Time namespaces: Docker 29.5.0 and later

Every container fails, GPUs or not:

```
docker run --runtime=sysbox-runc alpine echo ok
OCI runtime create failed: ... namespace {"time" ""} does not exist
```

The daemon now gives each container a private time namespace and writes it into the OCI spec; sysbox-runc does not know that namespace type and rejects the whole spec. Disabling CDI does not help here, and no sysbox version — 0.6.6 or 0.7.0 — accepts it. We first hit this on 29.6.2 in our own testing; a provider then confirmed both the error and the `time-namespaces` fix on 29.7.0.

If the error survives the `time-namespaces` setting, downgrade the daemon to **28.5.2**, which we verified working with containerd 2.2.6. The downgrade restarts the daemon too — stop any rentals first, same as above:

```bash
source /etc/os-release
sudo apt-get install -y --allow-downgrades \
  "docker-ce=5:28.5.2-1~ubuntu.${VERSION_ID}~${VERSION_CODENAME}" \
  "docker-ce-cli=5:28.5.2-1~ubuntu.${VERSION_ID}~${VERSION_CODENAME}"
sudo apt-mark hold docker-ce docker-ce-cli
sudo systemctl restart docker
```

If apt cannot find that version, list what your repo has: `apt-cache madison docker-ce`.

## Troubleshooting

- **`mount error: .../merged/proc/driver/nvidia: no such file or directory`** — sysbox is running without ID-mapped mounts. Usually the kernel is older than 5.19: see [Kernel 5.19+ is mandatory for GPUs](#kernel-519-is-mandatory-for-gpus). If `uname -r` already shows 5.19 or newer, the filesystem under Docker's data-root is the next suspect (`docker info --format '{{.Driver}}'` should say `overlay2`); `journalctl -u sysbox-mgr -b | grep -i id-mapped` shows what sysbox itself decided.
- **`namespace {"time" ""} does not exist`** — your Docker daemon is 29.5.0 or newer. See [Docker 29](#docker-29-two-required-settings).
- **`sysbox-runc not found`** — the installer didn't finish. Re-run `nvidia_docker_sysbox_setup.sh` and check its output.
- **GPU not visible inside the container** — confirm NVIDIA Container Toolkit is installed (`nvidia-container-cli --version`) and the Docker daemon was restarted after the install.
- **Validator still reports Sysbox missing** — wait one validation cycle (~15 min) and re-check from the Provider Portal.