Skip to main content

Sysbox

Sysbox is a container runtime that lets customers run Docker-in-Docker securely inside pods — without --privileged mode. Many workloads (custom image builds, CI/CD pipelines, system-level tooling) require it.

Sysbox is required

Every validator probes every node for the sysbox-runc runtime. Nodes missing it are rejected — they earn no emission and cannot be rented. Sysbox is a hard requirement for every Lium node, not an optional optimization. Setup takes ~5 minutes.

Requirements​

ComponentRequirement
Kernel5.19 or newer (6.x recommended). Older kernels cannot pass GPUs into a sysbox container — see below.
Docker27.x or 28.x works out of the box. 29.x needs daemon settings — see Docker 29.
Docker storage driveroverlay2 — check with docker info --format '{{.Driver}}'. On ZFS, and on some btrfs setups, the filesystem has no ID-map support and GPU passthrough breaks the same way an old kernel breaks it.
Architecturex86_64

Kernel 5.19+ is mandatory for GPUs​

Sysbox needs ID-mapped mounts, which the kernel supports for overlayfs only from 5.19 onward. On an older kernel — Ubuntu 22.04 ships 5.15 by default — sysbox falls back to shiftfs and runs the container rootfs from a different path (/var/lib/sysbox/shiftfs/<uuid>). The NVIDIA hook doesn't know about that move and still targets the original overlay2 path, which is not there in the container's mount namespace:

nvidia-container-cli: mount error: /var/lib/docker/overlay2/<id>/merged/proc/driver/nvidia: no such file or directory

Check your kernel:

uname -r

If it is below 5.19, install the HWE kernel and reboot:

sudo apt-get update
sudo apt-get install -y linux-generic-hwe-22.04
sudo reboot

Before you reboot:

  • Stop any rentals. Rebooting a node under an active rental costs you a penalty.
  • Check that the NVIDIA driver is DKMS-managed — dkms status should list an nvidia module. A driver installed from a .run file without DKMS will not load on the new kernel, and the node comes back with no GPUs at all. Reinstall the driver after the reboot if that happens.
  • The old kernel stays in the GRUB menu, so you can boot back into it if something goes wrong.
  • Ubuntu 20.04 has no upgrade path here — its HWE kernel is 5.15. You need 22.04 or newer.

After the reboot, confirm both uname -r and nvidia-smi before re-running the installer.

In our tests, upgrading the NVIDIA driver, the Container Toolkit, or Docker on its own did not fix this — only the kernel did.

Check the host first​

The installer runs a preflight before it installs anything. Run it on its own with --check — it installs and configures nothing (the Sysbox probe pulls alpine once) and prints one line per requirement:

curl -fsSL https://raw.githubusercontent.com/Datura-ai/lium-io/main/neurons/executor/nvidia_docker_sysbox_setup.sh | sudo bash -s -- --check
  …
  PASS Kernel 6.8.0-1063-aws (>= 5.19, ID-mapped mounts available).
  PASS Docker 29.8.0.
  …
  FIX  Kernel modules iptable_nat iptable_filter are not loaded — the validators' Docker-in-Docker probe needs legacy iptables and reports sysbox missing without them.
         sudo modprobe -a ip_tables iptable_nat iptable_filter   # -a: without it modprobe reads the 2nd and 3rd name as parameters of the 1st
         printf 'ip_tables\niptable_nat\niptable_filter\n' | sudo tee /etc/modules-load.d/lium-iptables.conf >/dev/null   # survives reboots
  …
  SKIP Disk >= 1.5x VRAM — no NVIDIA driver, VRAM unknown.
  PASS TCP 8080 (executor port) is free and not blocked by ufw.
  …
  Preflight: 6 PASS, 5 FIX, 1 SKIP.

Every FIX line is followed by the command that clears it. A SKIP is a check that could not run on this host — here the disk rule, because without the NVIDIA driver the VRAM is unknown; it counts neither way and clears once its input exists.

The checks:

  • root and x86_64;
  • kernel 5.19+;
  • Docker installed and running;
  • the NVIDIA driver loaded and at the validators' minimum version;
  • the legacy iptables modules the validators' Docker-in-Docker probe needs;
  • total disk of at least 1.5× total GPU VRAM on the filesystem under Docker's data-root — the idle-incentive rule;
  • the node's HTTP and SSH ports free and not blocked by ufw. The defaults are 8080 and 2200; for other ports put EXECUTOR_PORT=… SSH_PORT=… after sudo: … | sudo EXECUTOR_PORT=8001 SSH_PORT=2201 bash -s -- --check.

With --check it also reports the NVIDIA Container Toolkit, the Docker 29 settings, fusermount3 (from the fuse3 package) and whether a container starts under sysbox-runc. The command exits 1 while any line says FIX. Cloud firewalls sit outside the host: after the node is up, run nc -vz <public IP> 8080 2200 from another machine.

Without --check, a FIX on root, x86_64, the kernel or Docker stops the installer before it changes anything (Nothing was installed.); fix the lines and run it again. A FIX on the NVIDIA driver, the iptables modules, the disk or the ports is printed with its command and the install goes on, because a node without Sysbox is not listed at all. After SUCCESS (or Nothing to do. on a host that already has Sysbox 0.7.1 or newer; an older Sysbox is upgraded) a reminder says how many FIX lines are at the top and gives the --check command. Without a loaded NVIDIA driver the install still runs, but the final verification (nvidia-smi in a Sysbox container) fails: install the driver, reboot, and run the installer again.

Install​

The lium-io repo ships an installer that takes care of NVIDIA Container Toolkit + Sysbox in one go. From any Ubuntu host:

curl -fsSL https://raw.githubusercontent.com/Datura-ai/lium-io/main/neurons/executor/nvidia_docker_sysbox_setup.sh | sudo bash

The installer installs fuse3 together with nvidia-container-toolkit and jq, before the Sysbox package. Sysbox 0.7 mounts its FUSE file systems with fusermount3, which only fuse3 provides, and the sysbox-ce package does not pull it in. On Ubuntu 22.04, fuse3 replaces the older fuse package and still satisfies the sysbox-ce dependency.

nvidia-container-toolkit is not in the Ubuntu archive. The installer adds NVIDIA's apt repository itself (key in /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg, list in /etc/apt/sources.list.d/nvidia-container-toolkit.list) unless an apt source already points at nvidia.github.io/libnvidia-container. When an apt-get step fails, it prints the last 20 lines of apt's output and exits 1; fix what apt reports (a proxy, a held package, a broken source) and re-run — the installer is idempotent. A local lium-io clone from before 8 Sep 2026 still has the older installer, which hides apt's output and stops silently after [3/7] Installing packages; git pull first.

Or, if you already have the lium-io repo cloned locally:

cd lium-io/neurons/executor
chmod +x nvidia_docker_sysbox_setup.sh
sudo ./nvidia_docker_sysbox_setup.sh

Confirm /etc/docker/daemon.json includes the sysbox runtime:

{
  "runtimes": {
    "sysbox-runc": {
      "path": "/usr/bin/sysbox-runc"
    }
  }
}

Restart Docker:

sudo systemctl restart docker

Verify​

Run the same command our validator uses:

sudo docker run --rm --runtime=sysbox-runc --gpus all daturaai/compute-subnet-executor:latest nvidia-smi

If you see nvidia-smi output — you're good.

docker and sudo docker can reach different daemons

The installer above runs under sudo, so verify under sudo too. Rootless Docker, or a DOCKER_HOST set in your shell, points your plain docker client at another daemon. That daemon has no sysbox-runc and answers unknown or invalid runtime name: sysbox-runc, which looks like a failed install. Compare docker context ls with sudo docker context ls — contexts are per-user, so the two clients can differ.

Docker 29: two required settings​

DockerStatus
27.x, 28.xWorks as is. What we run and test against.
29.0–29.1Works — confirmed on 29.1.5. --gpus does not go through CDI until 29.2.0, and the installer sets cdi: false regardless.
29.2.x–29.4.xWorks with cdi off.
29.5.0 and laterWorks with cdi and time-namespaces off — confirmed by a provider on 29.7.0.

Docker 29 broke sysbox twice, in two unrelated ways. On 29.5.0 and later, turn both off with one block in /etc/docker/daemon.json:

{
  "features": {
    "cdi": false,
    "time-namespaces": false
  }
}

On 29.2.x–29.4.x, set only "cdi": false — the time-namespaces key appeared in 29.5 and is untested on older daemons.

Stop any rentals first. Restarting the daemon kills every running container on the host, and a pod killed under an active rental costs you a penalty. Then restart Docker and re-run the verify command:

sudo systemctl restart docker

Both settings are reversible — remove the features block and restart Docker to go back. cdi: false is daemon-wide, though: anything else on the host that gets its devices through CDI stops getting them that way. On a Lium node that is fine, since the NVIDIA legacy hook does the injection.

CDI: Docker 29.2.0 and later​

Sysbox + GPU fails with a permission denied error:

OCI runtime create failed: ... failed to open OCI spec file: ... permission denied

Docker 29.2 enables CDI (Container Device Interface) by default, routing --gpus through CDI — incompatible with sysbox's user namespace.

Time namespaces: Docker 29.5.0 and later​

Every container fails, GPUs or not:

docker run --runtime=sysbox-runc alpine echo ok
OCI runtime create failed: ... namespace {"time" ""} does not exist

The daemon now gives each container a private time namespace and writes it into the OCI spec; sysbox-runc before 0.7.1 does not know that namespace type and rejects the whole spec. Disabling CDI does not help here. Sysbox 0.6.6 and 0.7.0 reject it; 0.7.1 accepts it (we checked on Docker 29.8.1). The installer still turns time-namespaces off, so a host that keeps an older Sysbox also works. We first hit this on 29.6.2 in our own testing; a provider then confirmed both the error and the time-namespaces fix on 29.7.0.

If the error survives the time-namespaces setting, downgrade the daemon to 28.5.2, which we verified working with containerd 2.2.6. The downgrade restarts the daemon too — stop any rentals first, same as above:

source /etc/os-release
sudo apt-get install -y --allow-downgrades \
  "docker-ce=5:28.5.2-1~ubuntu.${VERSION_ID}~${VERSION_CODENAME}" \
  "docker-ce-cli=5:28.5.2-1~ubuntu.${VERSION_ID}~${VERSION_CODENAME}"
sudo apt-mark hold docker-ce docker-ce-cli
sudo systemctl restart docker

If apt cannot find that version, list what your repo has: apt-cache madison docker-ce.

Troubleshooting​

  • mount error: .../merged/proc/driver/nvidia: no such file or directory — sysbox is running without ID-mapped mounts. Usually the kernel is older than 5.19: see Kernel 5.19+ is mandatory for GPUs. If uname -r already shows 5.19 or newer, the filesystem under Docker's data-root is the next suspect (docker info --format '{{.Driver}}' should say overlay2); journalctl -u sysbox-mgr -b | grep -i id-mapped shows what sysbox itself decided.

  • setting up ID-mapped mount on path 212/fs (any number before /fs) — your host runs Sysbox 0.6.6 or older. With Docker's containerd image store, that version cannot start an image with about 44 or more layers; smaller images still start, so the node looks healthy until a large image arrives. Re-run nvidia_docker_sysbox_setup.sh when the node has no rentals (it upgrades Sysbox to 0.7.1), then check with sysbox-runc --version. If the output does not show Executor restarted, start the executor again with docker compose up -d from its folder.

  • namespace {"time" ""} does not exist — your Docker daemon is 29.5.0 or newer. See Docker 29.

  • failed to pre-register with sysbox-fs or fusermount: exec: "fusermount3": executable file not found in $PATH — the host has no fuse3, so every sysbox container fails to start even though the Sysbox package installed. An older installer can also end by blaming the Docker 29.5 time namespace, which is not the cause. Re-run nvidia_docker_sysbox_setup.sh (it installs fuse3 and reconfigures Sysbox; with a local clone, git pull first), or install it by hand with sudo apt-get install -y fuse3 and run the installer again.

  • sysbox-runc not found — the installer didn't finish. Re-run nvidia_docker_sysbox_setup.sh and check its output.

  • GPU not visible inside the container — confirm NVIDIA Container Toolkit is installed (nvidia-container-cli --version) and the Docker daemon was restarted after the install.

  • The verify command works but the validator still reports sysbox_runtime: false — the validator does not only start a sysbox container; it runs Docker inside it, and that inner dockerd uses legacy iptables. On a host whose netfilter is nftables-only (the default on Debian 13 and Ubuntu 26.04), the ip_tables kernel modules are not loaded, the inner daemon dies with can't initialize iptables table 'nat', sshd never starts, and the probe reports sysbox missing even though sysbox is fine. Load the three modules and make it stick across reboots — no Docker restart needed:

    sudo modprobe -a ip_tables iptable_nat iptable_filter
    printf 'ip_tables\niptable_nat\niptable_filter\n' | sudo tee /etc/modules-load.d/lium-iptables.conf
    lsmod | grep -E '^(ip_tables|iptable_nat|iptable_filter)'   # all three listed

    The next validation cycle (~15 min) passes.

  • Validator still reports Sysbox missing — wait one validation cycle (~15 min) and re-check from the Provider Portal. If it does not clear after two cycles and the verify command above works, see the nftables entry just above.