Furiosa SDK 2026.4: Enhanced MoE performance, hierarchical caching, expanded model coverage

Furiosa SDK 2026.4, our fourth major software release this year for RNGD, is now available via our Developer Portal. This release delivers more tokens per watt across a wide range of models, extends prefix caching beyond NPU memory, brings vision-language model support onto the standard scheduler, and completes our transition to simplified model loading via Furiosa Executable Bundles (FXB).

SDK 2026.4 expands the frontier of what enterprise customers can deploy on RNGD, which is now in mass production with TSMC. By drastically reducing cost-per-token and providing seamless drop-in software execution, we believe we offer the best solution for enterprises to deploy state-of-the-art AI workloads within their existing air-cooled data center infrastructure.

The sections below cover each change in greater detail. More documentation and instructions on upgrading are available here. You can find all change logs here.

If you are upgrading from 2026.3, please also read the 🚨 Breaking Changes & Deprecations section and the Upgrading FuriosaAI’s Software.

Hierarchical KV cache offloading

Prefix caching avoids repeated prefill for long, shared contexts, but a conventional cache is bounded by NPU DRAM. Once a cached prefix is evicted, a later request must recompute it even if the same prefix was used only moments earlier.

2026.4 introduces an experimental hierarchical KV cache that expands prefix reuse beyond NPU memory. This release also replaces separate global-only and hybrid prefix-cache implementations with a unified radix tree implementation. Global-attention and sliding-window KV caches share the same prefix-matching and scheduling infrastructure while retaining independent cache management.

The hierarchical cache extends this unified foundation across three tiers:

• L1 — NPU DRAM: the active, lowest-latency KV blocks used directly by attention.
• L2 — host memory: evicted or proactively backed-up blocks are copied over DMA and can be reloaded instead of recomputed.
• L3 — external storage: an optional backend retains prefixes beyond local process memory; a shared backend makes them reusable by other Furiosa-LLM replicas in the cluster.

L3 is storage-independent by design. A backend-agnostic KV cache connector interface separates cache management from backend-specific existence checks and data transfers. Additional storage systems can be integrated through this interface without changing the cache hierarchy. Mooncake is the first supported backend, providing content-addressed, cluster-wide prefix storage and asynchronous transfer outside the scheduler’s critical path.

Two write policies determine when KV cache is stored in L2:

• Write-through: copies a prefix to L2 once it has been reused a set number of times (default: 2)Keeping an L2 copy allows L1 blocks to be reclaimed without an eviction-time transfer. This suits deployments with enough NPU-to-host bandwidth and host-memory capacity to maintain copies in both tiers.
• Write-back: writes KV cache to L2 when it is evicted from L1. Deferring the copy reduces duplicate storage across the two tiers, making it useful when host-cache capacity is limited and more space is needed for evicted prefixes. The transfer can add eviction latency.

The two policies can be enabled together. With either one, the scheduler reuses the L1-resident portion of a matching prefix and reloads only its missing cached suffix. Offloading also works with hybrid global/sliding-window attention and intra-node pipeline parallelism. An experimental layerwise L2-to-L1 reload path can overlap host-to-NPU transfers with model execution; see Overlapping L2 Reload with Model Execution.

Enable the L2 tier by assigning a process-wide host-memory budget:

furiosa-llm serve <model> --experimental-kv-offload-host-memory-gb 64

Alternatively, --experimental-kv-offload-host-memory-ratio sizes L2 relative to each model lane’s L1 KV capacity. --experimental-kv-offload-write-through-threshold controls the promotion hit threshold (default: 2; 0 disables promotion), and --experimental-kv-offload-write-back enables the additional write-back policy. See Hierarchical Prefix Caching for policy and configuration details.

This feature is experimental and off by default. Its options are deliberately hidden from furiosa-llm serve --help while the policy and defaults are being tuned. With the backend included in 2026.4, L3 requires a compatible Mooncake deployment and shared-library installation; see Using Mooncake with Furiosa-LLM for setup and namespace configuration.

Optimized multimodal serving

The vision-language support introduced in 2026.3 now runs end to end on the regular FCFS scheduler. This unifies text-only and image-and-text requests under the same scheduling, prefix-cache, and data-parallel infrastructure instead of a separate proof-of-concept loop.

Multimodal request processing has been optimized in three areas:

• Batched vision encoding: images from multiple requests can be encoded together to improve throughput under concurrent load.
• Overlapped image preparation: image preparation on the host can overlap with NPU text generation for other requests, reducing NPU idle time.
• Image embedding reuse: cached image embeddings can be reused across conversation turns and data-parallel (DP) workers, avoiding repeated vision encoding for the same image.

These improvements apply to all currently supported vision-language models, including Gemma 4, Qwen3-VL, and EXAONE 4.5. See Vision-Language Models for details.

Expanded model and pooling coverage

2026.4 substantially broadens both generation and retrieval workloads:

• Qwen3-VL — new 2B and 4B Instruct/Thinking models, alongside the existing 32B family.
• Qwen3 Embedding — new 0.6B and 4B models, alongside the existing 8B model.
• Qwen3 Reranker — new 0.6B and 4B models, alongside the existing 8B model.
• Harrier OSS v1 — the 0.6B multilingual, instruction-aware text embedding model.
• Mistral NeMo — the 12B Mistral-Nemo-Instruct-2407-FP8 text-generation model.
• Llama 3.1 and Llama 3.3 — dynamically quantized FP8 variants.
• BGE-M3 — a multilingual dense-embedding model.
• BGE Reranker v2 M3 — a multilingual cross-encoder.
• E5-Mistral — the instruction-aware E5-Mistral-7B-Instruct embedding model.
• Gemma 4 — the 31B vision-language model supporting text and image input.
• EXAONE 4.5 — the 33B vision-language model supporting text and image input.

Consult Supported Models for the models that FuriosaAI publishes and validates.

Profile-Guided Decode Batching

Profile-guided decode batching uses actual NPU execution and I/O costs measured through profiling to optimize decode batches and bucket selection for requests with different context lengths. Originally introduced as an opt-in warm-up feature in 2025.3.0, it is now enabled by default for generative models in 2026.4.0 following performance improvements and stabilization.

According to GPT-OSS-120B benchmarks (TP=32, DP=1), enabling the feature increased ShareGPT output-token throughput by a median 22.7% and reduced mean time per output token (TPOT) by a median 16.0% across 20 matched configurations. These combine five concurrency levels (16, 32, 64, 128, and 256) with prefix caching on/off and overlap scheduling on/off (5 × 2 × 2). Each configuration compares the feature enabled and disabled under otherwise identical conditions. Results vary by workload.

On the first model load, profiling may take 1–2 minutes. Subsequent loads reuse compatible cached profiles. To disable profiling and profile-guided batching, pass --no-profile-guided-decode-batching to furiosa-llm serve. See Profile-Guided Decode Batching for configuration details.

Faster token sampling

Token sampling selects the next token from the logit scores produced by the model’s forward pass, using greedy selection or random sampling controlled by top_k and top_p. The selected token becomes the input to the next decode step, making sampling part of the critical path for time per output token (TPOT). Furiosa SDK 2026.4.0 introduces two major improvements to this process.

First, CPU sampling uses a faster algorithm for top-k and top-p filtering. Inspired by Qrita, it uses SIMD instructions to estimate the distribution of BF16 logits and narrow the candidate set before exact selection. It relaxes the threshold when necessary to retain enough candidates. Packed integer comparisons and partial selection further reduce top-p filtering overhead. These optimizations are enabled by default and require no additional configuration. Greedy sampling uses a separate argmax path.

In benchmarks using logits from GPT-OSS-20B and EXAONE 4.0 models, top-k/top-p filtering was 2.7–13.2× faster. In a separate ShareGPT benchmark with top_k=50 and top_p=0.9, output-token throughput improved by 6.9–16.9% and mean TPOT fell by 6.5–15.4% across concurrency levels from 16 to 256. Results depend on the model, logit distribution, and workload.

Second, NPU sampling runs the token-selection algorithm directly on the NPU. When no request in the batch needs logits on the host, this avoids transferring the full vocabulary logits to host memory. Batched NPU execution also parallelizes token selection across requests, reducing CPU sampling work for larger batches.

In a microbenchmark with Qwen3-MoE at batch size 128, NPU sampling path optimizations reduced the delay between the forward pass and NPU sampling from over 1 ms to about 200 µs, and host processing of sampled tokens from over 600 µs to about 100 µs per decode iteration.

Default FXB builds for all currently supported generative models include sampling kernels. NPU sampling is experimental and disabled by default. Enable it explicitly with:

furiosa-llm serve <model> --experimental-enable-npu-sampling

NPU sampling works with eager and overlap scheduling, subject to the model’s scheduling restrictions, and cannot be combined with speculative decoding. See Experimental NPU Sampling for configuration and Overlap Scheduling for compatibility.

Complete migration to FXB

Furiosa SDK 2026.4.0 removes support for the deprecated legacy artifact format. Compiled NPU kernels for all supported models are now provided through Furiosa Executable Bundles (FXB).

Introduced in 2026.3.0, FXB packages compiled kernels and their metadata in a single file, separate from model weights. Models can start without compiling kernels on the serving host, and bundles can be copied, cached, or published with a model. Compatible fine-tuned or weight-updated models can reuse the same FXB without recompiling.

We will continue to improve FXB tooling to make model compilation and the reuse and distribution of compiled kernels easier. See Furiosa Executable Bundles (FXB) for details on building, serving, and reusing bundles.

‍