Confidential Virtual Machine (CVM)

What is CVM and Why Do We Need It? (background — skip if you already know)

The Problem

In the current Lium platform design, GPU providers have full access to rental containers, which may allow a GPU provider to:

Inspect the contents of a rental container and exfiltrate sensitive data (model weights, proprietary code, private keys)
Modify or replace the node container with a customized version that behaves maliciously
Intercept network traffic or memory contents of running workloads
Tamper with job execution in ways that are invisible to validators and renters

This undermines trust in the platform — renters cannot be confident that their workloads are running in a safe, unmodified environment.

That access is a property of the design, not a permission: reaching into a rental workload is forbidden network-wide.

The Solution: CVM with Intel TDX

A Confidential Virtual Machine (CVM) is a hardware-isolated virtual machine powered by Intel Trust Domain Extensions (TDX). TDX provides:

Memory encryption — All VM memory is encrypted by the CPU hardware. The host OS and hypervisor cannot read VM memory contents, even with physical access.
Hardware attestation — The VM can generate cryptographically signed quotes (TDX quotes) that prove to a remote party exactly what software is running inside.
Measurement integrity — Any modification to the VM image or boot environment changes the hardware measurements, making tampering detectable.

How This Protects the Lium Platform

Threat	Without CVM	With CVM
GPU provider inspects rental container	Possible — full host access	Blocked — VM memory is hardware-encrypted
GPU provider replaces node container	Possible — container is writable from host	Blocked — TDX measurements detect any tampering
Rental container escapes to bare metal	Possible in edge cases	Blocked — rental container is nested inside TDX VM
Validator verifies node integrity	Not possible	Possible — TDX quotes provide cryptographic proof

In short: GPU providers can run the CVM but cannot see inside it. Renters are isolated from bare metal. Validators can cryptographically verify that the node is running exactly the expected software stack.

Flagship nodes: a CVM keeps the idle payout

An 8× H200, B200, or B300 node must offer GPU Profiling (ncu), GPU splitting, or confidential computing to earn the unrented incentive while idle. A node whose TDX quote the validator verifies satisfies that condition on its own — inside a CVM the host driver flag that GPU Profiling needs is out of reach anyway.

Prerequisites

1. CPU — Intel TDX Support

Your host machine must have a CPU with Intel TDX (Trust Domain Extensions) support.

Generation	Codename	TDX Support
4th Gen Intel Xeon Scalable	Sapphire Rapids	XCC and MCC SKUs only (not all SKUs)
5th Gen Intel Xeon Scalable	Emerald Rapids	All SKUs
Intel Xeon 6	Granite Rapids	All SKUs — also supports TDX Connect for encrypted CPU↔GPU communication

For 4th Gen, verify with your hardware vendor that the specific SKU supports TDX. TDX availability depends on both the CPU SKU and the platform firmware.

In addition to TDX, the CPU must support Intel SGX, as the key provider component relies on SGX enclaves. Verify that /dev/sgx_enclave and /dev/sgx_provision are present on the host after boot.

2. GPU — NVIDIA Confidential Compute Support

The GPU must support NVIDIA Confidential Computing. Supported architectures are Hopper (H100, H200) and Blackwell (B200, GB200).

For the full list of supported GPU SKUs, VBIOS versions, CUDA driver versions, and Confidential Computing modes, refer to the official NVIDIA Secure AI Compatibility Matrix.

Consumer-grade GPUs (RTX series, A-series workstation) do not support NVIDIA Confidential Computing.

3. BIOS / Firmware

Update to the latest BIOS from your server/motherboard vendor before enabling TDX
Enable the following settings in BIOS:
- Intel TDX (Trust Domain Extensions)
- Intel SGX (Software Guard Extensions)
- KVM / Virtualization (VT-x and VT-d)

Exact BIOS menu paths vary by vendor (Dell, HPE, Supermicro, etc.) — consult your server's platform configuration guide.

4. Kernel

TDX requires a kernel with Intel TDX support. Install the latest Intel-optimized kernel — the tested version is 6.14.0-1009-intel.

Check your current kernel version:

uname -r

The output should match or exceed the tested version, for example:

6.14.0-1009-intel

5. Operating System

Install Ubuntu 25.04 on the host machine
Ensure KVM is enabled and the kvm_intel module is loaded
Install Docker and Docker Compose

Check Before You Rack It

Five gates decide whether a node can ever attest. All five are checkable before the first CVM launch; found afterwards, each one has cost a provider a day or a full redeploy.

1. TDX Module Version

Intel's minimum for a TDX quote to verify is TDX module 2.0.08 (build 882, minor SVN 5). An older module is refused by every attester — the key provider, the validator, any third-party KMS.

On 6.x kernels the module prints its build number at boot:

sudo dmesg | grep -i 'TDX module'

The build number must be 882 or higher. Newer kernels print no build line; there the only proof is the tee_tcb_svn field of a TDX quote, whose first byte is the module's minor SVN and must be 05 or higher.

Symptom: the CVM boots, asks the key provider for its key, and the key provider logs No matching TCB level found.

Fix: a BIOS update from your server vendor, or — with no BIOS update — Intel's signed module dropped into the EFI partition. Download intel_tdx_module.tar.gz from the Intel TDX module releases, then:

tar -xvzf intel_tdx_module.tar.gz
sudo mkdir -p /boot/efi/EFI/TDX/
sudo cp TDX-Module/intel_tdx_module.so /boot/efi/EFI/TDX/TDX-SEAM.so
sudo cp TDX-Module/intel_tdx_module.so.sigstruct /boot/efi/EFI/TDX/TDX-SEAM.so.sigstruct
sudo reboot

If the firmware supports that path, the BIOS TDX loader picks the module up from the EFI partition at boot; if it does not, the module built into the BIOS keeps loading and nothing is lost. Only Intel-signed modules load, so a wrong file is inert. After the reboot, sudo dmesg | grep virt/tdx must still end in module initialized.

2. NUMA Topology

The firmware must not declare more NUMA nodes than it brings online:

cat /sys/devices/system/node/possible
cat /sys/devices/system/node/online

Both files must print the same range. When possible is wider than online — 0-19 against 0-1 on a Xeon 6767P, for example — the current key-provider image never loads its enclave. This is a property of the platform firmware, not of the kernel or the OS, so reinstalling either changes nothing.

Symptom: load_enclave ... EINVAL, raised before /dev/sgx_enclave is ever opened. The image is built on Gramine 1.5, which sizes its NUMA arrays from possible and then expects that many distance values per node.

Fix: change the base image in key-provider/Dockerfile.key-provider to Gramine 1.9 (gramineproject/gramine:1.9-jammy) and rebuild with sudo ./cvm_upgrade_guard.sh upgrade. The bug is fixed upstream from Gramine 1.6 onwards. A plain docker compose build in key-provider/ builds nothing since executor-v1.130, and the guard refuses the rebuild while any CVM disk exists on the host (see step 6).

3. Platform Registration and PCCS

A multi-package (multi-socket) platform must be registered with Intel before any PCCS can issue PCK certificates for it, and without those certificates DCAP has no collateral to verify the quote against. The default public PCCS cannot serve certificates for an unregistered platform.

Symptom: AESM error 44, then load_enclave ... EPERM.

Fix: register the platform, either by enabling SGX Auto MP Registration in BIOS or by running PCKIDRetrievalTool against a PCCS you operate with an Intel PCS API key:

PCKIDRetrievalTool -url https://localhost:8081 -use_secure_cert false

Then point pccs_url in key-provider/sgx_default_qcnl.conf at that PCCS — aesmd runs on the host network, so 127.0.0.1 there is the host itself.

4. QEMU Build

The host must run the dstack QEMU 9.2.1 build:

/opt/qemu-dstack/bin/qemu-system-x86_64 --version   # QEMU emulator version 9.2.1

The verifier reconstructs RTMR0 from an ACPI oracle built from that exact tree, so a host on distro QEMU 10.x can never pass — the oracle cannot forward-emulate newer ACPI tables.

Symptom: the node never validates, and the verifier replies RTMR0 mismatch.

Fix: build the dstack fork (kvinwang/qemu-tdx, branch dstack-qemu-9.2.1) with --prefix=/opt/qemu-dstack, then point the launcher at it in /etc/dstack/client.conf:

[qemu]
path = /opt/qemu-dstack/bin/qemu-system-x86_64

5. Runner Image Digest

EXECUTOR_RUNNER_IMAGE_DIGEST in .env is baked into the measured compose, so a wrong digest can only be corrected by recreating the CVM. Check it before you create one:

grep EXECUTOR_RUNNER_IMAGE_DIGEST .env

It must equal the digest published in the release notes of the release you are deploying — never one copied from an older .env, never one quoted in a chat.

From executor-v1.133 (21 Sep 2026) on, the release notes carry a section "CVM attestation" with two values:

The approved runner digest, the value for EXECUTOR_RUNNER_IMAGE_DIGEST.
The expected compose hash. This is the sha256 of app-compose.json a correct CVM produces; the validator accepts a CVM only when that hash is on the validator whitelist.

Releases before that one do not have this section. For executor-v1.128 through executor-v1.132 the values are the same as executor-v1.133's:

EXECUTOR_RUNNER_IMAGE_DIGEST=sha256:8c07d3a91f8900bd3f0e19025fb7a2c32f82577385550187b28c7769060d18b7
expected compose hash:       87d3430000bb7046a19eeaa6efe074fd8b5f7856bbfe2906addab238174c542d

A CVM created from executor-v1.108 through executor-v1.127 with the same digest measures 8224d58801af6333561f116e2d566b179b399f1d1d700f0e8a5ab9326ae901d9 (executor-v1.107 and earlier measure a third hash, 73ef4e1e…). The validator whitelist holds 8224d588… today; 87d34300… is added by the validator release that includes lium-io#1399 (merged 21 Sep 2026). Create the CVM with the values above and run it. Do not change the digest to chase 8224d588…: no digest produces it from the current files. If your release notes show other values, use those.

After sudo ./lium-cvm.sh new <name> and before run, check the measurement:

sha256sum run/vms/<name>/shared/app-compose.json

If the output is the expected compose hash, the CVM matches the release: delete nothing, run it. If it is not, the CVM does not match the release. Do not run a CVM that does not match the release. There are three causes: a wrong digest in .env, a changed measured file (app/docker-compose.yml, app/init_script.sh, app/pre_launch_script.sh), or a new flag that changes the measurement (--enable-logs, --enable-sysinfo, or an --env other than prod). Fix the cause, then recreate the VM directory and run new again.

warning

sudo rm -rf run/vms/<name> erases hda.img, the CVM's encrypted data disk, with every rental and log on it. A CVM you have just created with new and never run has no data yet. For any other CVM, export the data you need first; the disk cannot be read back after the directory is gone.

Symptom: the validator does not score the node, with no error on the node. A node that runs the retired July runner also refuses the validator with 401 Invalid validator signature.

Fix: run the sha256sum check above first. If the output is the expected compose hash, the CVM matches the release: delete nothing and look elsewhere. If it is not, recreate the CVM in this order. A node with a wrong digest still takes rentals in production, so stop with a pod running kills that pod: click Pause New Rentals for the node in the portal and wait for the last pod to end. Then correct the digest in .env, stop the CVM (sudo ./lium-cvm.sh stop <name>), export the data you need from it, remove the VM directory (sudo rm -rf run/vms/<name>, which erases hda.img), and recreate the CVM (sudo ./lium-cvm.sh new, then run). An existing CVM carries the old digest in its measurements; only new changes it.

Setup Guide

1. Check System Compatibility

Run the built-in compatibility check to verify that all required software and kernel features are present:

./lium-cvm.sh check

Fix any errors reported before continuing.

2. Check vfio-pci Module

The vfio-pci kernel module is required to pass GPUs through to the CVM. Verify it is loaded:

lsmod | grep vfio

If the module is not listed, load it manually:

sudo modprobe vfio-pci

To make this persistent across reboots, add it to /etc/modules:

echo "vfio-pci" | sudo tee -a /etc/modules

3. Verify GPU is Bound to vfio-pci

Confirm that the target GPU is using vfio-pci as its kernel driver. First, find the PCI address of your GPU:

lspci | grep -i h200   # adjust the model name as needed

Then inspect the specific device (replace 19:00.0 with your PCI address):

lspci -nnk -s 19:00.0

Expected output:

19:00.0 3D controller [0302]: NVIDIA Corporation GH100 [H200 SXM 141GB] [10de:2335] (rev a1)
    Subsystem: NVIDIA Corporation Device [10de:18be]
    Kernel driver in use: vfio-pci
    Kernel modules: nvidiafb, nouveau

The line Kernel driver in use: vfio-pci confirms the GPU is correctly bound. If it shows nvidia instead, the GPU is still claimed by the NVIDIA driver and must be rebound to vfio-pci before creating the CVM.

To list all available GPUs and their PCI addresses, you can also use:

./lium-cvm.sh lsgpu

4. Enable GPU Confidential Computing Mode

Use NVIDIA's gpu-admin-tools to configure Confidential Computing mode on the GPU.

git clone https://github.com/NVIDIA/gpu-admin-tools.git
cd gpu-admin-tools

Pick the case that matches your node:

Every mode switch triggers a GPU reset. Run these commands before the CVM is started. gpu-admin-tools works through sysfs, not through the NVIDIA driver, so the devices stay bound to vfio-pci.

On a node that already runs a CVM, stop it, switch the mode, then start it again. Do not recreate the VM directory with lium-cvm.sh new — that erases the encrypted data disk, and the data on it is unrecoverable by design.

Case 1 — Single Hopper GPU per CVM

Disable Protected PCIe mode first, then enable CC mode:

# Disable PPCIE mode
sudo python3 ./nvidia_gpu_tools.py --devices gpus --set-ppcie-mode=off --reset-after-ppcie-mode-switch

# Enable CC mode
sudo python3 ./nvidia_gpu_tools.py --devices gpus --set-cc-mode=on --reset-after-cc-mode-switch

Case 2 — All GPUs of an HGX Hopper node in one CVM (8×H100 / 8×H200)

Use Protected PCIe (PPCIE) mode instead of CC mode. It is the only Hopper mode that puts several GPUs into one CVM; GPU-to-GPU traffic then goes over NVLink only. Per-GPU CC mode on an 8× board isolates each GPU and the driver disables NVLink — nvidia-smi topo -m shows PHB instead of NV18.

Protected PCIe takes the whole board: all GPUs and all NVSwitches of the node go to the same CVM, a subset is not supported. In step 7 set CVM_GPUS=all.

The NVSwitches must be bound to vfio-pci like the GPUs: add their device ID to vfio-pci.ids (10de:22a3 on HGX H100/H200) and check them the same way as in step 3 — lspci -nnk -d 10de: | grep -A2 Bridge must show Kernel driver in use: vfio-pci for every switch. lsgpu does not list them.

Prerequisites: HGX firmware 1.7.1 or newer, Secure Boot off, IOMMU on.

Disable CC mode first, then enable Protected PCIe mode on the GPUs and NVSwitches:

# Disable CC mode (GPUs only; NVSwitches have no CC mode)
sudo python3 ./nvidia_gpu_tools.py --devices gpus --set-cc-mode=off --reset-after-cc-mode-switch

# Enable PPCIE mode on GPUs and all NVSwitches
sudo python3 ./nvidia_gpu_tools.py --devices gpus,nvswitches --set-ppcie-mode=on --reset-after-ppcie-mode-switch

# Verify all devices (8 GPUs + 4 NVSwitches on an HGX board) before starting the CVM
sudo python3 ./nvidia_gpu_tools.py --devices gpus,nvswitches --query-ppcie-mode

Once the CVM is up, verify from a pod rented on this node (the host gets no shell into the guest):

nvidia-smi conf-compute --get-multigpu-mode   # expected: Protected PCIe
nvidia-smi topo -m                            # expected: NV18 between all GPUs

Optional deeper check — attest the GPUs and NVSwitches with NVIDIA's verifier:

# Install and run as the same user: a user-site install is invisible to sudo python3.
# 2.x needs NVIDIA's nvattest CLI and a different invocation; this is the 1.x procedure
sudo python3 -m pip install "nv-ppcie-verifier>=1.0,<2.0"

# The wheel ships no console script, hence python3 -m. Expected result: PASS
sudo python3 -m ppcie.verifier.verification --gpu-attestation-mode=REMOTE --switch-attestation-mode=REMOTE

warning

nvidia-smi conf-compute -f prints CC status: OFF when Protected PCIe is correctly on. That line is not the signal — --get-multigpu-mode is.

Case 3 — Blackwell (B200 / B300)

Protected PCIe does not exist on Blackwell (--set-ppcie-mode is Hopper-only). Blackwell encrypts NVLink itself, so CC mode on the GPUs is the whole setup:

sudo python3 ./nvidia_gpu_tools.py --devices gpus --set-cc-mode=on --reset-after-cc-mode-switch

Lium has not verified multi-GPU confidential computing on Blackwell.

5. Download the OS Image

Download the dstack TDX OS image that the CVM will boot from:

./lium-cvm.sh download

The image is saved to run/images/ and reused on subsequent runs.

6. Start the Key Provider

The key-provider is an SGX enclave service that supplies sealing keys to the TDX VM. It must be running on the host before the CVM boots.

The service consists of two containers: aesmd (Intel SGX architectural enclave service) and gramine-sealing-key-provider (the key provider itself), listening on 127.0.0.1:3443.

Start the containers through the upgrade guard, from the same directory as lium-cvm.sh:

sudo ./cvm_upgrade_guard.sh start

On a host with no CVM disk, the guard builds the key provider image and pins its image id in /var/lib/lium-cvm/key-provider.image. Every later start runs exactly that pinned image and builds nothing. The reason: a rebuilt key provider has a new MRENCLAVE, and every existing CVM data disk on the host becomes unreadable at its next boot (lium-io#1366, executor-v1.130). Since that release, a hand-run docker compose up or docker compose build in key-provider/ builds nothing: the compose file there has no build: section.

Verify both containers are running (stay in the same directory; the compose project is the one the guard started):

docker compose -f key-provider/docker-compose.yaml ps

Check the key-provider logs for errors:

docker compose -f key-provider/docker-compose.yaml logs -f gramine-sealing-key-provider

Confirm the endpoint is reachable:

curl -k https://localhost:3443

Note: lium-cvm.sh run also starts the key provider through the guard, on the pinned image. Running the guard yourself first is still recommended: on a fresh host you see the one build and any SGX error before the CVM boots.

7. Configure Environment and Create the CVM

Copy the example environment file and fill in your settings:

cp .env.example .env

Key fields to configure:

# Provider identity
MINER_HOTKEY_SS58_ADDRESS=<your_hotkey>

# Ports
SSH_PORT=2200
RENTING_PORT_RANGE="19001,19002,19003"

# CVM resources
CVM_VCPUS=16
CVM_MEMORY=64G
CVM_DISK=200G

# GPU passthrough — use PCI addresses from step 3, or "all" to pass through every GPU and NVSwitch
CVM_GPUS=19:00.0,3b:00.0
# CVM_GPUS=all

Important: CVM_GPUS must list the PCI addresses of GPUs that are already bound to vfio-pci (verified in step 3). Using the wrong address or a GPU still bound to the NVIDIA driver will cause the CVM to fail on launch. A Protected PCIe node (step 4, case 2) must use CVM_GPUS=all: a list passes only the named GPUs and leaves the NVSwitches on the host.

Once .env is configured, create the CVM. new and run need sudo since executor-v1.130: they take the host lock /var/lock/lium-cvm.lock and register the VM directory in /var/lib/lium-cvm/vm-dirs, so the guard's disk inventory sees it.

sudo ./lium-cvm.sh new my-executor
sha256sum run/vms/my-executor/shared/app-compose.json

The sha256sum line must print the expected compose hash from the release notes. If your release notes have no "CVM attestation" section, take the value from Runner Image Digest. That section also says what to do when the value differs.

8. Run the CVM

Start the CVM:

sudo ./lium-cvm.sh run my-executor

To verify the launch command without actually starting the VM, use the dry-run flag:

sudo ./lium-cvm.sh run my-executor --dry-run

On startup QEMU may print:

qemu-system-x86_64: warning: TDX doesn't support requested feature: CPUID.07H_01H:EDX.avx10 [bit 19]

This is harmless: the host CPU advertises AVX10, the TDX module masks it, and QEMU only reports the mask. CPUID bits are not part of RTMR0, so attestation is unaffected.

9. Check the Dashboard

Once the CVM is running, a logging dashboard is available at:

http://<host-ip>:8090

Port 8090 must be included in RENTING_PORT_RANGE in your .env file so it is exposed by the CVM:

RENTING_PORT_RANGE="8090,19001,19002,19003"

The dashboard provides real-time logs and status for the node running inside the CVM.

Troubleshooting

CVM fails to start — vfio-dev: No such file or directory

Symptoms that trace back to one of the gates in Check Before You Rack It:

Symptom	Cause	Where to fix
Key provider logs `No matching TCB level found`	TDX module older than 2.0.08	TDX Module Version
`load_enclave ... EINVAL`	firmware declares more possible than online NUMA nodes	NUMA Topology
AESM error 44, then `load_enclave ... EPERM`	platform not registered, or the PCCS has no PCK certificates for it	Platform Registration and PCCS
Node never validates, verifier replies `RTMR0 mismatch`	host on distro QEMU 10.x instead of the dstack 9.2.1 build	QEMU Build
Node refuses the validator with `401 Invalid validator signature`	stale `EXECUTOR_RUNNER_IMAGE_DIGEST`	Runner Image Digest
Node never scores, no error in the node log	compose hash of `app-compose.json` not on the validator whitelist (wrong digest, edited measured file, or a `new` flag)	Runner Image Digest

1. CVM fails to start — `vfio-dev: No such file or directory`

Symptom

QEMU exits immediately with an error like:

qemu-system-x86_64: -device vfio-pci,host=19:00.0,...: vfio 0000:19:00.0:
vfio /sys/bus/pci/devices/0000:19:00.0/vfio-dev: couldn't open directory
/sys/bus/pci/devices/0000:19:00.0/vfio-dev: No such file or directory

Cause

The kernel was not booted with the parameters required to enable Intel IOMMU and bind the GPU to vfio-pci at boot time. Without intel_iommu=on, the kernel does not create the vfio-dev sysfs entry even if vfio-pci is loaded.

Solution

Find the PCI device IDs of your NVIDIA GPUs:

lspci -nn | grep -i nvidia

Example output:

19:00.0 3D controller [0302]: NVIDIA Corporation GH100 [H200 SXM 141GB] [10de:2335] (rev a1)
3b:00.0 3D controller [0302]: NVIDIA Corporation GH100 [H200 SXM 141GB] [10de:22a3] (rev a1)

Note the IDs in brackets — e.g. 10de:2335,10de:22a3.

Edit the GRUB configuration at /etc/default/grub:

sudo nano /etc/default/grub

Set the following lines (replace vfio-pci.ids with the IDs from step 1):

GRUB_CMDLINE_LINUX_DEFAULT="kvm_intel.tdx=on nohibernate intel_iommu=on video=efifb:off vfio_iommu_type1.dma_entry_limit=1048576 vfio-pci.ids=10de:2335,10de:22a3 kvm_intel.tdx=1 default_hugepagesz=1G hugepagesz=1G hugepages=10"
GRUB_CMDLINE_LINUX="console=tty0"

Apply the changes and reboot:

sudo update-grub
sudo reboot

After reboot, verify the GPU is bound to vfio-pci (see Setup Guide step 3).

Prerequisites​

1. CPU — Intel TDX Support​

2. GPU — NVIDIA Confidential Compute Support​

3. BIOS / Firmware​

4. Kernel​

5. Operating System​

Check Before You Rack It​

1. TDX Module Version​

2. NUMA Topology​

3. Platform Registration and PCCS​

4. QEMU Build​

5. Runner Image Digest​

Setup Guide​

1. Check System Compatibility​

2. Check vfio-pci Module​

3. Verify GPU is Bound to vfio-pci​

4. Enable GPU Confidential Computing Mode​

5. Download the OS Image​

6. Start the Key Provider​

7. Configure Environment and Create the CVM​

8. Run the CVM​

9. Check the Dashboard​

Troubleshooting​

1. CVM fails to start — vfio-dev: No such file or directory​