---
sidebar_position: 6
---
> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lium.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Confidential Virtual Machine (CVM)
**What is CVM and Why Do We Need It?** (background — skip if you already know)
**The Problem**
In the current Lium platform design, GPU providers have full access to rental containers, which may allow a GPU provider to:
- Inspect the contents of a rental container and exfiltrate sensitive data (model weights, proprietary code, private keys)
- Modify or replace the node container with a customized version that behaves maliciously
- Intercept network traffic or memory contents of running workloads
- Tamper with job execution in ways that are invisible to validators and renters
This undermines trust in the platform — renters cannot be confident that their workloads are running in a safe, unmodified environment.
That access is a property of the design, not a permission: reaching into a rental workload is [forbidden network-wide](../rental-privacy.md).
**The Solution: CVM with Intel TDX**
A **Confidential Virtual Machine (CVM)** is a hardware-isolated virtual machine powered by [Intel Trust Domain Extensions (TDX)](https://www.intel.com/content/www/us/en/developer/tools/trust-domain-extensions/overview.html). TDX provides:
- **Memory encryption** — All VM memory is encrypted by the CPU hardware. The host OS and hypervisor cannot read VM memory contents, even with physical access.
- **Hardware attestation** — The VM can generate cryptographically signed quotes (TDX quotes) that prove to a remote party exactly what software is running inside.
- **Measurement integrity** — Any modification to the VM image or boot environment changes the hardware measurements, making tampering detectable.
**How This Protects the Lium Platform**
| Threat | Without CVM | With CVM |
|---|---|---|
| GPU provider inspects rental container | Possible — full host access | Blocked — VM memory is hardware-encrypted |
| GPU provider replaces node container | Possible — container is writable from host | Blocked — TDX measurements detect any tampering |
| Rental container escapes to bare metal | Possible in edge cases | Blocked — rental container is nested inside TDX VM |
| Validator verifies node integrity | Not possible | Possible — TDX quotes provide cryptographic proof |
In short: GPU providers can run the CVM but cannot see inside it. Renters are isolated from bare metal. Validators can cryptographically verify that the node is running exactly the expected software stack.
:::tip Flagship nodes: a CVM keeps the idle payout
An 8× H200, B200, or B300 node must offer [GPU Profiling (ncu)](./gpu-profiling.md), [GPU splitting](./gpu-splitting.md), or confidential computing to earn the [unrented incentive](../rewards/emission.mdx#unrented-pool) while idle. A node whose TDX quote the validator verifies satisfies that condition on its own — inside a CVM the host driver flag that GPU Profiling needs is out of reach anyway.
:::
---
## Prerequisites
### 1. CPU — Intel TDX Support
Your host machine must have a CPU with [Intel TDX (Trust Domain Extensions)](https://cc-enabling.trustedservices.intel.com/intel-tdx-enabling-guide/01/introduction/) support.
| Generation | Codename | TDX Support |
|---|---|---|
| 4th Gen Intel Xeon Scalable | Sapphire Rapids | XCC and MCC SKUs only (not all SKUs) |
| 5th Gen Intel Xeon Scalable | Emerald Rapids | All SKUs |
| Intel Xeon 6 | Granite Rapids | All SKUs — also supports [TDX Connect](https://community.intel.com/t5/Blogs/Tech-Innovation/Data-Center/Announcing-Intel-TDX-Connect-Support-on-Intel-Xeon-6/post/1668423) for encrypted CPU↔GPU communication |
> For 4th Gen, verify with your hardware vendor that the specific SKU supports TDX. TDX availability depends on both the CPU SKU and the platform firmware.
In addition to TDX, the CPU must support **Intel SGX**, as the key provider component relies on SGX enclaves. Verify that `/dev/sgx_enclave` and `/dev/sgx_provision` are present on the host after boot.
### 2. GPU — NVIDIA Confidential Compute Support
The GPU must support [NVIDIA Confidential Computing](https://www.nvidia.com/en-us/data-center/solutions/confidential-computing/). Supported architectures are **Hopper** (H100, H200) and **Blackwell** (B200, GB200).
For the full list of supported GPU SKUs, VBIOS versions, CUDA driver versions, and Confidential Computing modes, refer to the official [NVIDIA Secure AI Compatibility Matrix](https://www.nvidia.com/en-us/data-center/solutions/confidential-computing/secure-ai-compatibility-matrix/).
> Consumer-grade GPUs (RTX series, A-series workstation) do **not** support NVIDIA Confidential Computing.
### 3. BIOS / Firmware
1. Update to the **latest BIOS** from your server/motherboard vendor before enabling TDX
2. Enable the following settings in BIOS:
- **Intel TDX** (Trust Domain Extensions)
- **Intel SGX** (Software Guard Extensions)
- **KVM / Virtualization** (VT-x and VT-d)
> Exact BIOS menu paths vary by vendor (Dell, HPE, Supermicro, etc.) — consult your server's platform configuration guide.
### 4. Kernel
TDX requires a kernel with Intel TDX support. Install the latest Intel-optimized kernel — the tested version is `6.14.0-1009-intel`.
Check your current kernel version:
```bash
uname -r
```
The output should match or exceed the tested version, for example:
```
6.14.0-1009-intel
```
### 5. Operating System
1. Install **[Ubuntu 25.04](https://releases.ubuntu.com/25.04/)** on the host machine
2. Ensure KVM is enabled and the `kvm_intel` module is loaded
3. Install Docker and Docker Compose
---
## Check Before You Rack It
Five gates decide whether a node can ever attest. All five are checkable before the first CVM launch; found afterwards, each one has cost a provider a day or a full redeploy.
### 1. TDX Module Version
Intel's minimum for a TDX quote to verify is **TDX module 2.0.08** (build 882, minor SVN 5). An older module is refused by every attester — the key provider, the validator, any third-party KMS.
On 6.x kernels the module prints its build number at boot:
```bash
sudo dmesg | grep -i 'TDX module'
```
The build number must be **882 or higher**. Newer kernels print no build line; there the only proof is the `tee_tcb_svn` field of a TDX quote, whose first byte is the module's minor SVN and must be `05` or higher.
**Symptom:** the CVM boots, asks the key provider for its key, and the key provider logs `No matching TCB level found`.
**Fix:** a BIOS update from your server vendor, or — with no BIOS update — Intel's signed module dropped into the EFI partition. Download `intel_tdx_module.tar.gz` from the [Intel TDX module releases](https://github.com/intel/confidential-computing.tdx.tdx-module/releases), then:
```bash
tar -xvzf intel_tdx_module.tar.gz
sudo mkdir -p /boot/efi/EFI/TDX/
sudo cp TDX-Module/intel_tdx_module.so /boot/efi/EFI/TDX/TDX-SEAM.so
sudo cp TDX-Module/intel_tdx_module.so.sigstruct /boot/efi/EFI/TDX/TDX-SEAM.so.sigstruct
sudo reboot
```
If the firmware supports that path, the BIOS TDX loader picks the module up from the EFI partition at boot; if it does not, the module built into the BIOS keeps loading and nothing is lost. Only Intel-signed modules load, so a wrong file is inert. After the reboot, `sudo dmesg | grep virt/tdx` must still end in `module initialized`.
---
### 2. NUMA Topology
The firmware must not declare more NUMA nodes than it brings online:
```bash
cat /sys/devices/system/node/possible
cat /sys/devices/system/node/online
```
Both files must print the same range. When `possible` is wider than `online` — `0-19` against `0-1` on a Xeon 6767P, for example — the current key-provider image never loads its enclave. This is a property of the platform firmware, not of the kernel or the OS, so reinstalling either changes nothing.
**Symptom:** `load_enclave ... EINVAL`, raised before `/dev/sgx_enclave` is ever opened. The image is built on Gramine 1.5, which sizes its NUMA arrays from `possible` and then expects that many distance values per node.
**Fix:** rebuild the key provider in the `key-provider/` directory on a Gramine 1.9 base image (`gramineproject/gramine:1.9-jammy`) — the bug is fixed upstream from Gramine 1.6 onwards.
---
### 3. Platform Registration and PCCS
A multi-package (multi-socket) platform must be registered with Intel before any PCCS can issue PCK certificates for it, and without those certificates DCAP has no collateral to verify the quote against. The default public PCCS cannot serve certificates for an unregistered platform.
**Symptom:** AESM error 44, then `load_enclave ... EPERM`.
**Fix:** register the platform, either by enabling **SGX Auto MP Registration** in BIOS or by running `PCKIDRetrievalTool` against a PCCS you operate with an Intel PCS API key:
```bash
PCKIDRetrievalTool -url https://localhost:8081 -use_secure_cert false
```
Then point `pccs_url` in `key-provider/sgx_default_qcnl.conf` at that PCCS — `aesmd` runs on the host network, so `127.0.0.1` there is the host itself.
---
### 4. QEMU Build
The host must run the **dstack QEMU 9.2.1** build:
```bash
/opt/qemu-dstack/bin/qemu-system-x86_64 --version # QEMU emulator version 9.2.1
```
The verifier reconstructs RTMR0 from an ACPI oracle built from that exact tree, so a host on distro QEMU 10.x can never pass — the oracle cannot forward-emulate newer ACPI tables.
**Symptom:** the node never validates, and the verifier replies `RTMR0 mismatch`.
**Fix:** build the dstack fork ([`kvinwang/qemu-tdx`](https://github.com/kvinwang/qemu-tdx), branch `dstack-qemu-9.2.1`) with `--prefix=/opt/qemu-dstack`, then point the launcher at it in `/etc/dstack/client.conf`:
```ini
[qemu]
path = /opt/qemu-dstack/bin/qemu-system-x86_64
```
---
### 5. Runner Image Digest
`EXECUTOR_RUNNER_IMAGE_DIGEST` in `.env` is baked into the measured compose, so a wrong digest can only be corrected by recreating the CVM. Check it before you create one:
```bash
grep EXECUTOR_RUNNER_IMAGE_DIGEST .env
```
It must equal the digest published in the release notes of the release you are deploying — never one copied from an older `.env`, never one quoted in a chat.
**Symptom:** the node refuses the validator with `401 Invalid validator signature`, so validation never completes.
**Fix:** correct the digest in `.env` and recreate the CVM (`./lium-cvm.sh new`, then `run`); an existing CVM carries the old digest in its measurements.
---
## Setup Guide
### 1. Check System Compatibility
Run the built-in compatibility check to verify that all required software and kernel features are present:
```bash
./lium-cvm.sh check
```
Fix any errors reported before continuing.
---
### 2. Check vfio-pci Module
The `vfio-pci` kernel module is required to pass GPUs through to the CVM. Verify it is loaded:
```bash
lsmod | grep vfio
```
If the module is not listed, load it manually:
```bash
sudo modprobe vfio-pci
```
To make this persistent across reboots, add it to `/etc/modules`:
```bash
echo "vfio-pci" | sudo tee -a /etc/modules
```
---
### 3. Verify GPU is Bound to vfio-pci
Confirm that the target GPU is using `vfio-pci` as its kernel driver. First, find the PCI address of your GPU:
```bash
lspci | grep -i h200 # adjust the model name as needed
```
Then inspect the specific device (replace `19:00.0` with your PCI address):
```bash
lspci -nnk -s 19:00.0
```
Expected output:
```
19:00.0 3D controller [0302]: NVIDIA Corporation GH100 [H200 SXM 141GB] [10de:2335] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:18be]
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau
```
The line `Kernel driver in use: vfio-pci` confirms the GPU is correctly bound. If it shows `nvidia` instead, the GPU is still claimed by the NVIDIA driver and must be rebound to `vfio-pci` before creating the CVM.
To list all available GPUs and their PCI addresses, you can also use:
```bash
./lium-cvm.sh lsgpu
```
---
### 4. Enable GPU Confidential Computing Mode
Use NVIDIA's [gpu-admin-tools](https://github.com/NVIDIA/gpu-admin-tools) to configure Confidential Computing mode on the GPU.
```bash
git clone https://github.com/NVIDIA/gpu-admin-tools.git
cd gpu-admin-tools
```
Pick the case that matches your node:
> Every mode switch triggers a GPU reset. Run these commands before the CVM is started. gpu-admin-tools works through sysfs, not through the NVIDIA driver, so the devices stay bound to `vfio-pci`.
>
> On a node that already runs a CVM, stop it, switch the mode, then start it again. Do not recreate the VM directory with `lium-cvm.sh new` — that erases the encrypted data disk, and the data on it is unrecoverable by design.
**Case 1 — Single Hopper GPU per CVM**
Disable Protected PCIe mode first, then enable CC mode:
```bash
# Disable PPCIE mode
sudo python3 ./nvidia_gpu_tools.py --devices gpus --set-ppcie-mode=off --reset-after-ppcie-mode-switch
# Enable CC mode
sudo python3 ./nvidia_gpu_tools.py --devices gpus --set-cc-mode=on --reset-after-cc-mode-switch
```
**Case 2 — All GPUs of an HGX Hopper node in one CVM (8×H100 / 8×H200)**
Use Protected PCIe (PPCIE) mode instead of CC mode. It is the only Hopper mode that puts several GPUs into one CVM; GPU-to-GPU traffic then goes over NVLink only. Per-GPU CC mode on an 8× board isolates each GPU and the driver disables NVLink — `nvidia-smi topo -m` shows `PHB` instead of `NV18`.
Protected PCIe takes the whole board: all GPUs and all NVSwitches of the node go to the same CVM, a subset is not supported. In step 7 set `CVM_GPUS=all`.
The NVSwitches must be bound to `vfio-pci` like the GPUs: add their device ID to `vfio-pci.ids` (`10de:22a3` on HGX H100/H200) and check them the same way as in step 3 — `lspci -nnk -d 10de: | grep -A2 Bridge` must show `Kernel driver in use: vfio-pci` for every switch. `lsgpu` does not list them.
Prerequisites: HGX firmware 1.7.1 or newer, Secure Boot off, IOMMU on.
Disable CC mode first, then enable Protected PCIe mode on the GPUs and NVSwitches:
```bash
# Disable CC mode (GPUs only; NVSwitches have no CC mode)
sudo python3 ./nvidia_gpu_tools.py --devices gpus --set-cc-mode=off --reset-after-cc-mode-switch
# Enable PPCIE mode on GPUs and all NVSwitches
sudo python3 ./nvidia_gpu_tools.py --devices gpus,nvswitches --set-ppcie-mode=on --reset-after-ppcie-mode-switch
# Verify all devices (8 GPUs + 4 NVSwitches on an HGX board) before starting the CVM
sudo python3 ./nvidia_gpu_tools.py --devices gpus,nvswitches --query-ppcie-mode
```
Once the CVM is up, verify from a pod rented on this node (the host gets no shell into the guest):
```bash
nvidia-smi conf-compute --get-multigpu-mode # expected: Protected PCIe
nvidia-smi topo -m # expected: NV18 between all GPUs
```
Optional deeper check — attest the GPUs and NVSwitches with NVIDIA's verifier:
```bash
# Install and run as the same user: a user-site install is invisible to sudo python3.
# 2.x needs NVIDIA's nvattest CLI and a different invocation; this is the 1.x procedure
sudo python3 -m pip install "nv-ppcie-verifier>=1.0,<2.0"
# The wheel ships no console script, hence python3 -m. Expected result: PASS
sudo python3 -m ppcie.verifier.verification --gpu-attestation-mode=REMOTE --switch-attestation-mode=REMOTE
```
:::warning
`nvidia-smi conf-compute -f` prints `CC status: OFF` when Protected PCIe is correctly on. That line is not the signal — `--get-multigpu-mode` is.
:::
**Case 3 — Blackwell (B200 / B300)**
Protected PCIe does not exist on Blackwell (`--set-ppcie-mode` is Hopper-only). Blackwell encrypts NVLink itself, so CC mode on the GPUs is the whole setup:
```bash
sudo python3 ./nvidia_gpu_tools.py --devices gpus --set-cc-mode=on --reset-after-cc-mode-switch
```
Lium has not verified multi-GPU confidential computing on Blackwell.
---
### 5. Download the OS Image
Download the dstack TDX OS image that the CVM will boot from:
```bash
./lium-cvm.sh download
```
The image is saved to `run/images/` and reused on subsequent runs.
---
### 6. Start the Key Provider
The **key-provider** is an SGX enclave service that supplies sealing keys to the TDX VM. It must be running on the host before the CVM boots.
The service consists of two containers: `aesmd` (Intel SGX architectural enclave service) and `gramine-sealing-key-provider` (the key provider itself), listening on `127.0.0.1:3443`.
1. Navigate to the key-provider directory and start the containers:
```bash
cd key-provider && docker compose up --build -d
```
2. Verify both containers are running:
```bash
docker compose ps
```
3. Check the key-provider logs for errors:
```bash
docker compose logs -f gramine-sealing-key-provider
```
4. Confirm the endpoint is reachable:
```bash
curl -k https://localhost:3443
```
> **Note:** `lium-cvm.sh run` will also attempt to auto-start the key-provider, but starting it manually first is recommended so you can catch build errors early — especially on first setup when the container images need to be built.
---
### 7. Configure Environment and Create the CVM
Copy the example environment file and fill in your settings:
```bash
cp .env.example .env
```
Key fields to configure:
```bash
# Provider identity
MINER_HOTKEY_SS58_ADDRESS=
# Ports
SSH_PORT=2200
RENTING_PORT_RANGE="19001,19002,19003"
# CVM resources
CVM_VCPUS=16
CVM_MEMORY=64G
CVM_DISK=200G
# GPU passthrough — use PCI addresses from step 3, or "all" to pass through every GPU and NVSwitch
CVM_GPUS=19:00.0,3b:00.0
# CVM_GPUS=all
```
> **Important:** `CVM_GPUS` must list the PCI addresses of GPUs that are already bound to `vfio-pci` (verified in step 3). Using the wrong address or a GPU still bound to the NVIDIA driver will cause the CVM to fail on launch. A Protected PCIe node (step 4, case 2) must use `CVM_GPUS=all`: a list passes only the named GPUs and leaves the NVSwitches on the host.
Once `.env` is configured, create the CVM:
```bash
./lium-cvm.sh new my-executor
```
---
### 8. Run the CVM
Start the CVM:
```bash
./lium-cvm.sh run my-executor
```
To verify the launch command without actually starting the VM, use the dry-run flag:
```bash
./lium-cvm.sh run my-executor --dry-run
```
On startup QEMU may print:
```
qemu-system-x86_64: warning: TDX doesn't support requested feature: CPUID.07H_01H:EDX.avx10 [bit 19]
```
This is harmless: the host CPU advertises AVX10, the TDX module masks it, and QEMU only reports the mask. CPUID bits are not part of RTMR0, so attestation is unaffected.
---
### 9. Check the Dashboard
Once the CVM is running, a logging dashboard is available at:
```
http://:8090
```
Port `8090` must be included in `RENTING_PORT_RANGE` in your `.env` file so it is exposed by the CVM:
```bash
RENTING_PORT_RANGE="8090,19001,19002,19003"
```
The dashboard provides real-time logs and status for the node running inside the CVM.
---
## Troubleshooting
1. [CVM fails to start — `vfio-dev: No such file or directory`](#1-cvm-fails-to-start--vfio-dev-no-such-file-or-directory)
Symptoms that trace back to one of the gates in [Check Before You Rack It](#check-before-you-rack-it):
| Symptom | Cause | Where to fix |
|---|---|---|
| Key provider logs `No matching TCB level found` | TDX module older than 2.0.08 | [TDX Module Version](#1-tdx-module-version) |
| `load_enclave ... EINVAL` | firmware declares more possible than online NUMA nodes | [NUMA Topology](#2-numa-topology) |
| AESM error 44, then `load_enclave ... EPERM` | platform not registered, or the PCCS has no PCK certificates for it | [Platform Registration and PCCS](#3-platform-registration-and-pccs) |
| Node never validates, verifier replies `RTMR0 mismatch` | host on distro QEMU 10.x instead of the dstack 9.2.1 build | [QEMU Build](#4-qemu-build) |
| Node refuses the validator with `401 Invalid validator signature` | stale `EXECUTOR_RUNNER_IMAGE_DIGEST` | [Runner Image Digest](#5-runner-image-digest) |
---
### 1. CVM fails to start — `vfio-dev: No such file or directory`
**Symptom**
QEMU exits immediately with an error like:
```
qemu-system-x86_64: -device vfio-pci,host=19:00.0,...: vfio 0000:19:00.0:
vfio /sys/bus/pci/devices/0000:19:00.0/vfio-dev: couldn't open directory
/sys/bus/pci/devices/0000:19:00.0/vfio-dev: No such file or directory
```
**Cause**
The kernel was not booted with the parameters required to enable Intel IOMMU and bind the GPU to `vfio-pci` at boot time. Without `intel_iommu=on`, the kernel does not create the `vfio-dev` sysfs entry even if `vfio-pci` is loaded.
**Solution**
1. Find the PCI device IDs of your NVIDIA GPUs:
```bash
lspci -nn | grep -i nvidia
```
Example output:
```
19:00.0 3D controller [0302]: NVIDIA Corporation GH100 [H200 SXM 141GB] [10de:2335] (rev a1)
3b:00.0 3D controller [0302]: NVIDIA Corporation GH100 [H200 SXM 141GB] [10de:22a3] (rev a1)
```
Note the IDs in brackets — e.g. `10de:2335,10de:22a3`.
2. Edit the GRUB configuration at `/etc/default/grub`:
```bash
sudo nano /etc/default/grub
```
Set the following lines (replace `vfio-pci.ids` with the IDs from step 1):
```
GRUB_CMDLINE_LINUX_DEFAULT="kvm_intel.tdx=on nohibernate intel_iommu=on video=efifb:off vfio_iommu_type1.dma_entry_limit=1048576 vfio-pci.ids=10de:2335,10de:22a3 kvm_intel.tdx=1 default_hugepagesz=1G hugepagesz=1G hugepages=10"
GRUB_CMDLINE_LINUX="console=tty0"
```
3. Apply the changes and reboot:
```bash
sudo update-grub
sudo reboot
```
4. After reboot, verify the GPU is bound to `vfio-pci` (see [Setup Guide step 3](#3-verify-gpu-is-bound-to-vfio-pci)).