Skip to main content

GPU clock lock

Lock your pod's GPU clocks to a fixed MHz range, for example for repeatable benchmarks. The pod keeps running, and the clocks go back to default when the pod is deleted.

Lock the clocks​

Open it

On the pod page of a running pod, click Lock GPU clocks.

Set the range

Enter Min MHz and Max MHz, tick the GPUs (on a multi-GPU pod), and click Lock clocks. The node uses the closest range the GPU supports and shows the applied range.

Reset the clocks​

Open Lock GPU clocks again and click Reset to default.

Use the CLI or API​

lium gpu-clocks my-pod --lock 1400,1800           # all GPUs
lium gpu-clocks my-pod --lock 1400,1800 --gpu 0   # GPU 0 only
lium gpu-clocks my-pod --reset
curl -X POST https://lium.io/api/pods/<pod_id>/gpu-clocks \
  -H "X-API-Key: $LIUM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"min_mhz": 1400, "max_mhz": 1800}'

Send {"reset": true} to reset. Add "gpu_indices": [0] for some GPUs only (indices as nvidia-smi shows them in the pod). The SDK has Lium.lock_gpu_clocks(pod, min_mhz, max_mhz) and Lium.reset_gpu_clocks(pod).

If it fails​

The error names its reason:

  • clock_lock_not_supported: this machine or GPU cannot lock clocks.
  • mig_enabled: the GPU is in MIG mode.
  • operation_in_progress: another GPU lock or reset is running on the pod. Wait for it.
  • gpu_clocks_unavailable: no answer in time. Check nvidia-smi in the pod before trying again.
Last updated on