Sysbox

Sysbox is a container runtime that lets customers run Docker-in-Docker securely inside pods — without --privileged mode. Many workloads (custom image builds, CI/CD pipelines, system-level tooling) require it.

Sysbox is required

Every validator probes every node for the sysbox-runc runtime. Nodes missing it are rejected — they earn no emission and cannot be rented. Sysbox is a hard requirement for every Lium node, not an optional optimization. Setup takes ~5 minutes.

Requirements

Component	Requirement
Kernel	5.19 or newer (6.x recommended). Older kernels cannot pass GPUs into a sysbox container — see below.
Docker	27.x or 28.x works out of the box. 29.x needs daemon settings — see Docker 29.
Docker storage driver	overlay2 — check with docker info --format '{{.Driver}}'. On ZFS, and on some btrfs setups, the filesystem has no ID-map support and GPU passthrough breaks the same way an old kernel breaks it.
Architecture	x86_64

Kernel 5.19+ is mandatory for GPUs

Sysbox needs ID-mapped mounts, which the kernel supports for overlayfs only from 5.19 onward. On an older kernel — Ubuntu 22.04 ships 5.15 by default — sysbox falls back to shiftfs and runs the container rootfs from a different path (/var/lib/sysbox/shiftfs/<uuid>). The NVIDIA hook doesn't know about that move and still targets the original overlay2 path, which is not there in the container's mount namespace:

nvidia-container-cli: mount error: /var/lib/docker/overlay2/<id>/merged/proc/driver/nvidia: no such file or directory

Check your kernel:

uname -r

If it is below 5.19, install the HWE kernel and reboot:

sudo apt-get update
sudo apt-get install -y linux-generic-hwe-22.04
sudo reboot

Before you reboot:

• Stop any rentals. Rebooting a node under an active rental costs you a penalty.
• Check that the NVIDIA driver is DKMS-managed — dkms status should list an nvidia module. A driver installed from a .run file without DKMS will not load on the new kernel, and the node comes back with no GPUs at all. Reinstall the driver after the reboot if that happens.
• The old kernel stays in the GRUB menu, so you can boot back into it if something goes wrong.
• Ubuntu 20.04 has no upgrade path here — its HWE kernel is 5.15. You need 22.04 or newer.

After the reboot, confirm both uname -r and nvidia-smi before re-running the installer.

In our tests, upgrading the NVIDIA driver, the Container Toolkit, or Docker on its own did not fix this — only the kernel did.

Install

The lium-io repo ships an installer that takes care of NVIDIA Container Toolkit + Sysbox in one go. From any Ubuntu host:

curl -fsSL https://raw.githubusercontent.com/Datura-ai/lium-io/main/neurons/executor/nvidia_docker_sysbox_setup.sh | sudo bash

Or, if you already have the lium-io repo cloned locally:

cd lium-io/neurons/executor
chmod +x nvidia_docker_sysbox_setup.sh
sudo ./nvidia_docker_sysbox_setup.sh

Confirm /etc/docker/daemon.json includes the sysbox runtime:

{
  "runtimes": {
    "sysbox-runc": {
      "path": "/usr/bin/sysbox-runc"
    }
  }
}

Restart Docker:

sudo systemctl restart docker

Verify

Run the same command our validator uses:

docker run --rm --runtime=sysbox-runc --gpus all daturaai/compute-subnet-executor:latest nvidia-smi

If you see nvidia-smi output — you're good.

Docker 29: two required settings

Docker	Status
27.x, 28.x	Works as is. What we run and test against.
29.0–29.1	Untested.
29.2.x–29.4.x	Works with cdi off.
29.5.0 and later	Works with cdi and time-namespaces off — confirmed by a provider on 29.7.0.

Docker 29 broke sysbox twice, in two unrelated ways. On 29.5.0 and later, turn both off with one block in /etc/docker/daemon.json:

{
  "features": {
    "cdi": false,
    "time-namespaces": false
  }
}

On 29.2.x–29.4.x, set only "cdi": false — the time-namespaces key appeared in 29.5 and is untested on older daemons.

Stop any rentals first. Restarting the daemon kills every running container on the host, and a pod killed under an active rental costs you a penalty. Then restart Docker and re-run the verify command:

sudo systemctl restart docker

Both settings are reversible — remove the features block and restart Docker to go back. cdi: false is daemon-wide, though: anything else on the host that gets its devices through CDI stops getting them that way. On a Lium node that is fine, since the NVIDIA legacy hook does the injection.

CDI: Docker 29.2.0 and later

Sysbox + GPU fails with a permission denied error:

OCI runtime create failed: ... failed to open OCI spec file: ... permission denied

Docker 29.2 enables CDI (Container Device Interface) by default, routing --gpus through CDI — incompatible with sysbox's user namespace.

Time namespaces: Docker 29.5.0 and later

Every container fails, GPUs or not:

docker run --runtime=sysbox-runc alpine echo ok
OCI runtime create failed: ... namespace {"time" ""} does not exist

The daemon now gives each container a private time namespace and writes it into the OCI spec; sysbox-runc does not know that namespace type and rejects the whole spec. Disabling CDI does not help here, and no sysbox version — 0.6.6 or 0.7.0 — accepts it. We first hit this on 29.6.2 in our own testing; a provider then confirmed both the error and the time-namespaces fix on 29.7.0.

If the error survives the time-namespaces setting, downgrade the daemon to 28.5.2, which we verified working with containerd 2.2.6. The downgrade restarts the daemon too — stop any rentals first, same as above:

source /etc/os-release
sudo apt-get install -y --allow-downgrades \
  "docker-ce=5:28.5.2-1~ubuntu.${VERSION_ID}~${VERSION_CODENAME}" \
  "docker-ce-cli=5:28.5.2-1~ubuntu.${VERSION_ID}~${VERSION_CODENAME}"
sudo apt-mark hold docker-ce docker-ce-cli
sudo systemctl restart docker

If apt cannot find that version, list what your repo has: apt-cache madison docker-ce.

Troubleshooting

• mount error: .../merged/proc/driver/nvidia: no such file or directory — sysbox is running without ID-mapped mounts. Usually the kernel is older than 5.19: see Kernel 5.19+ is mandatory for GPUs. If uname -r already shows 5.19 or newer, the filesystem under Docker's data-root is the next suspect (docker info --format '{{.Driver}}' should say overlay2); journalctl -u sysbox-mgr -b | grep -i id-mapped shows what sysbox itself decided.
• namespace {"time" ""} does not exist — your Docker daemon is 29.5.0 or newer. See Docker 29.
• sysbox-runc not found — the installer didn't finish. Re-run nvidia_docker_sysbox_setup.sh and check its output.
• GPU not visible inside the container — confirm NVIDIA Container Toolkit is installed (nvidia-container-cli --version) and the Docker daemon was restarted after the install.
• Validator still reports Sysbox missing — wait one validation cycle (~15 min) and re-check from the Provider Portal.