Serverless functions with @lium.machine
@lium.machine runs a Python function on a rented GPU pod and removes the pod after. You call it like any other function.
import lium
@lium.machine(machine="RTX4090")
def matmul(n: int) -> float:
import torch
a = torch.randn(n, n, device="cuda")
return float((a @ a).sum())
print(matmul(4096))
[lium] matmul: renting 1xRTX4090 $0.30/h (swift-fox-c8, Romania), removal in 1.2h
[lium] matmul: pod ready in 38s
[lium] matmul: running
[lium] matmul: done in 71s (~$0.0059)
[lium] matmul: pod removed
-1342.5
Sign in once with lium init (or set LIUM_API_KEY).
How a call works
- Pick a node.
machineis"<count>x<gpu>"or"<gpu>", for example"1xH200"or"RTX4090". The cheapest node with exactly that many GPUs is rented. - Cap the cost. The pod is scheduled for removal at
timeout + 15 min(timeoutdefaults to one hour). - Ship the function. Only the function's own
defis sent. Arguments go with it. - Install requirements.
requirementsare installed once per pod, on top of the image's torch. - Run. Everything the function prints shows in your terminal.
- Return or raise. You get the result, or the same exception type the function raised.
- Remove the pod. Or keep it for the next call with
keep_warm.
What a call costs
Measured on a 1×RTX 4090 node at $0.30/h:
| wall time | cost | |
|---|---|---|
| cold call (rent, boot, run) | ~70 s | ~$0.006 |
| call on a warm pod | ~19 s | ~$0.002 |
A warm pod bills while it waits. Set keep_warm to the gap between your calls, and call close() when you are done.
Example 1: a GPU operation on the image's torch
No requirements: the default template already has torch with CUDA.
import lium
@lium.machine(machine="RTX4090")
def bench(n: int) -> dict:
import time
import torch
a = torch.randn(n, n, device="cuda", dtype=torch.float16)
torch.cuda.synchronize()
t = time.perf_counter()
for _ in range(10):
a @ a
torch.cuda.synchronize()
ms = (time.perf_counter() - t) * 100
tflops = 2 * n ** 3 / (ms / 1000) / 1e12
print(f"{torch.cuda.get_device_name(0)}: {ms:.1f} ms per {n}x{n} fp16 matmul")
return {"device": torch.cuda.get_device_name(0), "ms": round(ms, 1), "tflops": round(tflops, 1)}
print(bench(8192))
Return plain Python types. Turn tensors into .tolist() or .cpu().numpy().
Example 2: transformers inference, with requirements
Torch comes from the image. transformers and accelerate are installed once per pod.
import lium
@lium.machine(machine="RTX4090", requirements=["transformers", "accelerate"], timeout=600)
def generate(prompt: str, max_new_tokens: int = 64) -> str:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "HuggingFaceTB/SmolLM2-135M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="cuda")
messages = [{"role": "user", "content": prompt}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False,
pad_token_id=tokenizer.eos_token_id)
return tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True).strip()
try:
print(generate("In one sentence, who discovered penicillin?"))
except lium.RemoteExecutionError as e: # a timeout, a killed process, or a non-builtin remote exception
print(e, e.remote_traceback)
except Exception as e: # a builtin remote exception, e.g. OSError for a bad model id
print(type(e).__name__, e)
if isinstance(e.__cause__, lium.RemoteExecutionError):
print(e.__cause__.remote_traceback) # what happened on the pod
Example 3: a warm pod, a batch, and a local escape hatch
keep_warm=600 keeps the pod ten minutes after each call. .map() runs every item on one pod. .close() removes the pod.
import lium
@lium.machine(machine="RTX4090", requirements=["sentence-transformers"], keep_warm=600)
def embed(texts: list[str]) -> list[list[float]]:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2", device="cuda")
return model.encode(texts, normalize_embeddings=True).tolist()
batches = [["a red apple", "a green pear"], ["a fast car"], ["an old boat", "a new ship"]]
vectors = embed.map(batches) # one pod, three calls
print(len(vectors), len(vectors[0][0])) # 3 384
embed.close() # remove the warm pod now
For tests, embed.local(texts) runs the function on your machine.
What can and cannot travel
- Arguments: anything that can be pickled.
- Results: plain Python types (
None,bool,int,float,str,bytes, lists, tuples, sets, dicts),datetimetypes,Decimal,Path,UUID, and numpy arrays. Anything else raiseslium.ResultEncodingError. - Code: only the function's
deftravels. Import inside the function and pass other values as arguments. - Errors: a builtin exception (
ValueError,OSError…) comes back with its own type. Anything else, or a timeout, raiseslium.RemoteExecutionError. - Ctrl-C removes the pod.
Parameters
| parameter | default | meaning |
|---|---|---|
machine | - | "<count>x<gpu>" or "<gpu>"; cheapest matching node |
requirements | None | pip packages, installed once per pod |
template_id | node default | Docker template to rent with |
timeout | 3600 | seconds the function may run; None for no limit |
keep_warm | 0 | seconds the pod stays after a call, billed while idle |
cleanup | True | False leaves the pod running after the call (remove it with lium rm). The removal scheduled at rent time still stands: cancel it with lium schedules rm if the pod must outlive it |
local | False | run in this process; LIUM_MACHINE_LOCAL=1 does it for every function |
quiet | False | no [lium] progress lines on stderr |
Methods on the decorated function: f.remote(*args) (same as f(*args)), f.local(*args), f.map(iterable), f.close().