Skip to main content

Serverless functions with @lium.machine

@lium.machine runs a Python function on a rented GPU pod and removes the pod after. You call it like any other function.

import lium

@lium.machine(machine="RTX4090")
def matmul(n: int) -> float:
import torch
a = torch.randn(n, n, device="cuda")
return float((a @ a).sum())

print(matmul(4096))
[lium] matmul: renting 1xRTX4090 $0.30/h (swift-fox-c8, Romania), removal in 1.2h
[lium] matmul: pod ready in 38s
[lium] matmul: running
[lium] matmul: done in 71s (~$0.0059)
[lium] matmul: pod removed
-1342.5

Sign in once with lium init (or set LIUM_API_KEY).

How a call works​

  1. Pick a node. machine is "<count>x<gpu>" or "<gpu>", for example "1xH200" or "RTX4090". The cheapest node with exactly that many GPUs is rented.
  2. Cap the cost. The pod is scheduled for removal at timeout + 15 min (timeout defaults to one hour).
  3. Ship the function. Only the function's own def is sent. Arguments go with it.
  4. Install requirements. requirements are installed once per pod, on top of the image's torch.
  5. Run. Everything the function prints shows in your terminal.
  6. Return or raise. You get the result, or the same exception type the function raised.
  7. Remove the pod. Or keep it for the next call with keep_warm.

What a call costs​

Measured on a 1×RTX 4090 node at $0.30/h:

wall timecost
cold call (rent, boot, run)~70 s~$0.006
call on a warm pod~19 s~$0.002

A warm pod bills while it waits. Set keep_warm to the gap between your calls, and call close() when you are done.

Example 1: a GPU operation on the image's torch​

No requirements: the default template already has torch with CUDA.

import lium

@lium.machine(machine="RTX4090")
def bench(n: int) -> dict:
import time
import torch
a = torch.randn(n, n, device="cuda", dtype=torch.float16)
torch.cuda.synchronize()
t = time.perf_counter()
for _ in range(10):
a @ a
torch.cuda.synchronize()
ms = (time.perf_counter() - t) * 100
tflops = 2 * n ** 3 / (ms / 1000) / 1e12
print(f"{torch.cuda.get_device_name(0)}: {ms:.1f} ms per {n}x{n} fp16 matmul")
return {"device": torch.cuda.get_device_name(0), "ms": round(ms, 1), "tflops": round(tflops, 1)}

print(bench(8192))

Return plain Python types. Turn tensors into .tolist() or .cpu().numpy().

Example 2: transformers inference, with requirements​

Torch comes from the image. transformers and accelerate are installed once per pod.

import lium

@lium.machine(machine="RTX4090", requirements=["transformers", "accelerate"], timeout=600)
def generate(prompt: str, max_new_tokens: int = 64) -> str:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "HuggingFaceTB/SmolLM2-135M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="cuda")
messages = [{"role": "user", "content": prompt}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False,
pad_token_id=tokenizer.eos_token_id)
return tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True).strip()

try:
print(generate("In one sentence, who discovered penicillin?"))
except lium.RemoteExecutionError as e: # a timeout, a killed process, or a non-builtin remote exception
print(e, e.remote_traceback)
except Exception as e: # a builtin remote exception, e.g. OSError for a bad model id
print(type(e).__name__, e)
if isinstance(e.__cause__, lium.RemoteExecutionError):
print(e.__cause__.remote_traceback) # what happened on the pod

Example 3: a warm pod, a batch, and a local escape hatch​

keep_warm=600 keeps the pod ten minutes after each call. .map() runs every item on one pod. .close() removes the pod.

import lium

@lium.machine(machine="RTX4090", requirements=["sentence-transformers"], keep_warm=600)
def embed(texts: list[str]) -> list[list[float]]:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2", device="cuda")
return model.encode(texts, normalize_embeddings=True).tolist()

batches = [["a red apple", "a green pear"], ["a fast car"], ["an old boat", "a new ship"]]
vectors = embed.map(batches) # one pod, three calls
print(len(vectors), len(vectors[0][0])) # 3 384

embed.close() # remove the warm pod now

For tests, embed.local(texts) runs the function on your machine.

What can and cannot travel​

  • Arguments: anything that can be pickled.
  • Results: plain Python types (None, bool, int, float, str, bytes, lists, tuples, sets, dicts), datetime types, Decimal, Path, UUID, and numpy arrays. Anything else raises lium.ResultEncodingError.
  • Code: only the function's def travels. Import inside the function and pass other values as arguments.
  • Errors: a builtin exception (ValueError, OSError …) comes back with its own type. Anything else, or a timeout, raises lium.RemoteExecutionError.
  • Ctrl-C removes the pod.

Parameters​

parameterdefaultmeaning
machine-"<count>x<gpu>" or "<gpu>"; cheapest matching node
requirementsNonepip packages, installed once per pod
template_idnode defaultDocker template to rent with
timeout3600seconds the function may run; None for no limit
keep_warm0seconds the pod stays after a call, billed while idle
cleanupTrueFalse leaves the pod running after the call (remove it with lium rm). The removal scheduled at rent time still stands: cancel it with lium schedules rm if the pod must outlive it
localFalserun in this process; LIUM_MACHINE_LOCAL=1 does it for every function
quietFalseno [lium] progress lines on stderr

Methods on the decorated function: f.remote(*args) (same as f(*args)), f.local(*args), f.map(iterable), f.close().