Build at any layer, from model to silicon
Serving
furiosa-llm: drop-in, OpenAI-compatible inference
pip install furiosa-llm
furiosa-llm serve [model]
# Serving on :8000 (OpenAI-compatible)from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1"
)
client.chat.completions.create(model="[model]", ...)from furiosa_llm import LLM
llm = LLM(
model="[model]",
dtype="[precision]",
)
out = llm.generate(prompts)Start from a model that already runs
gpt-oss
20B · 120B
FP8
Text generation
Qwen3
4B · 8B · 32B
FP8
Text generation
Solar Open
100B
NVFp4A16
Text generation
4B · 8B · 32B
FP8
Text Generation
30B-A3B
FP8
Text Generation
30B-A3B
FP8
Code Generation
2B · 4B · 32B
BF16
Vision-Language
0.6B · 4B · 8B
BF16
Embedding
0.6B · 4B · 8B
BF16
Reranking
20B · 120B
MXFP4
Text Generation
31B
FP8
Vision-Language
100B
NVFP4A16
Text Generation
236B-A23B
NVFP4A16
Text Generation
33B
FP8
Vision-Language
32B
FP8
Text Generation
7.8B · 32B
BF16
Text Generation
12B
FP8
Text Generation
70B
FP8 · INT8 · BF16
Text Generation
8B
FP8 · BF16
Text Generation
7B · 8B · 14B · 32B · 70B
BF16
Reasoning
32B
BF16
Reasoning
0.5B · 7B · 14B · 32B
BF16
Text Generation
7B · 14B · 32B
BF16
Code Generation
0.6B
BF16
Embedding
0.6B
FP32
Reranking
7B
FP16
Embedding
0.6B
BF16
Embedding
27B · 35B-A3B
FP8
Vision-Language
0.8B
BF16
Vision-Language
Showing 6 of 32
DEPLOYING
Pick the deployment path that fits your needs
A dedicated NXT RNGD Server, hosted at a data center near you.
Call models served on RNGD through our cloud partners. Nothing to install.
PROGRAMMING
Move your model to RNGD with furiosa-torch
import torch, furiosa.torch
device = torch.device("furiosa:0")
model = MyModel().to(device)out = model(x.to(device)) # eager: ops run as dispatched
compiled = torch.compile(model, backend=furiosa.torch.backend)
out = compiled(x.to(device)) # whole-graph compile
runnable = CompileModule.from_module(model, inputs) # AOT → RunnableTwo levels of kernel programming
TCL
Declare what to compute over named symbolic axes. Kernels are generic over shape; the compiler owns tiling, layout and scheduling.
import furiosa.tcl as tcl
@tcl.kernel
def linear(x: tcl.bf16[M, K], w: tcl.bf16[N, K]) -> tcl.f32[M, N]:
# axes missing from the output are contracted away
return tcl.operation.contract(x, w, out=[M, N])vISA
Hand-optimize the hot path. Reason in tensors while directly managing memory placement and Tensor Unit scheduling.
#[device(chip=1)]
fn gemm(a: DmTensor<bf16, m![M, K]>,
b: DmTensor<bf16, m![K, N]>) -> DmTensor<f32, m![M, N]> {
// you choose the memory tier and the schedule
a.fetch().contract(b).commit()
}Dive Deeper
