TRT-LLM vs vLLM vs llama.cpp on Blackwell — sparkinfer v0.3.7 owns decode on the RTX 5090

I develop and maintain sparkinfer — a Blackwell-native inference runtime that is ~30% faster than llama.cpp on decode while keeping the same LLM quality (~64.4% vs ~65.9% overall on a 196-item benchmark; within ~1.5 points). On an RTX 5090 with Qwen3-30B-A3B at Q4_K_M, that’s ~485 tok/s vs ~366 tok/s. That’s the headline.

sparkinfer: ~30% faster decode than llama.cpp, same LLM quality — on a 2.5 MB native binary.

Below that, I also ran TRT-LLM, vLLM, and llama.cpp head-to-head on my RTX PRO 6000 Blackwell — same model (Qwen3-30B-A3B), same 28K-token input — to show where the datacenter engines win (prefill) and where llama.cpp wins on my workstation card (decode vs TRT-LLM). Different GPU, different phase, different answer.

Table of contents

The setup

• GPU: NVIDIA RTX PRO 6000 Blackwell (GB202, SM 12.0, 95 GB GDDR7, 188 SMs, ~1.79 TB/s bandwidth) — the same card I put the zk-STARK prover on.
• Model: Qwen3-30B-A3B — Qwen3MoeForCausalLM, 48 layers, 8 of 128 experts per token (6.25% activation), 40,960 native context, BF16 weights at 57 GB.
• Engines: TRT-LLM 1.2.1 (PyTorch backend), vLLM 0.23.0, llama.cpp (BF16 / Q8_0 / Q4_K_M).
• Workload: 28,002-token input → 512-token output, batch size 1, greedy (temperature 0), 3 runs averaged, after a warmup.

A methodology note up front, because honesty matters more than a clean story: my prompt builder capped at ~28K tokens, so the “32K” and “128K” tests used the same input content (the 128K run only differed in having YaRN active). And TRT-LLM ran on its PyTorch (_torch) backend, not the compiled TRT engine path — the engine path would likely add 10–25%. So read these as a real but specific snapshot, not a final verdict.

The head-to-head (32K context, 28,002 → 512 tokens)

Two different engines hold the two crowns. That’s the whole story, and it isn’t an accident.

The split: prefill is compute, decode is bandwidth

TTFT (prefill) — TRT-LLM wins, 4.5–9.2× over llama.cpp. Processing a 28K-token prompt is compute-bound: you run the whole context through the model in big parallel matmuls. TRT-LLM’s fused MoE GEMM and graph-optimized prefill deliver 779 ms; vLLM is close at 909 ms; llama.cpp lands at 3.5–7.2 s. If your product shows the user a first word fast (interactive chat), this is the axis that matters and the datacenter engines own it.

Decode — llama.cpp Q4_K_M wins, 2× over TRT-LLM. Autoregressive decode at batch 1 is the opposite regime: it’s memory-bandwidth-bound. Every single token, you stream the active weights out of VRAM to produce one output token; the tensor cores mostly idle. So the metric that decides decode speed isn’t FLOPs — it’s bytes moved per token. 4-bit weights are 4× smaller than BF16, so Q4_K_M moves ~4× less data per step and hits 351.6 tok/s, more than double TRT-LLM’s 166.8. The roofline literally flips the winner between the two phases.

That’s the lesson I keep coming back to: “which engine is fastest?” is the wrong question. “Fastest for which phase, at which batch size?” is the right one. Prefill-heavy (RAG, long prompts, short answers) and decode-heavy (long-form generation) workloads want different engines on the same hardware.

The counterintuitive bit: BF16 prefill is slower than Q8/Q4

Look at llama.cpp’s prompt-processing speeds: BF16 does 3,894 tok/s, but Q8_0 does 8,061 and Q4_K_M does 7,838. Full precision is slower at prefill — which surprises people, because prefill is “compute-bound.” The reason is that MoE prefill is still gated by getting the expert weights to the tensor cores, and BF16 experts are 2× the bytes of Q8_0. Even in the compute-bound phase, the bandwidth to feed the cores is the bottleneck, so the more tightly packed weights keep the cores better fed. Quantization helps on both sides of the prefill/decode divide here.

sparkinfer v0.3.7: decode king on the RTX 5090

sparkinfer v0.3.7 on the RTX 5090 — the numbers behind the ~30% claim. The PRO 6000 table earlier is a separate snapshot: TRT-LLM owns prefill there, llama.cpp Q4_K_M owns decode vs the datacenter engines.

Where TRT-LLM and vLLM retrofit sm_120, sparkinfer targets Blackwell natively — ~15 s compile, decode-only today, experts kept quantized-resident and dequantized on read. On v0.3.7 (Jul 7) it ships two models and leads llama.cpp on both — same RTX 5090, same Q4_K_M GGUF, warm and interleaved vs llama-bench:

That’s not a marginal win — it’s +32% on the same model this post benchmarks, and +38% on Qwen3.6-35B-A3B, a Gated-DeltaNet hybrid llama.cpp hasn’t really optimized yet. sparkinfer shipped Qwen3.6 cold and still pulled ahead: 23 → 387 tok/s in 48 hours, driven by a shared-expert coalesced-GEMV kernel (7.2× on that path), int8 dp4a MMVQ for Q5_K expert down-projection, split-K bf16 dense projection GEMV, and Q8 shared-expert MMVQ with MoE pipelining. (The split-K-for-occupancy lever is the same one I used for decode attention in genie-ai-runtime.)

Runtime footprint

Speed isn’t the only axis where a from-scratch engine can differ. Runtime footprint, excluding model weights and Python launcher scripts:

A 2.5 MB native binary vs 3.6 GB for TensorRT-LLM is a different product shape entirely — closer to shipping a single executable on edge hardware than deploying a Python framework stack.

Quality check

Decode speed means nothing if the model stops being the model. LLM quality check, 25% benchmark tier, 196 items:

Honest read: sparkinfer trails llama.cpp by 1.5 points overall and vLLM AWQ by 2.6 points — not identical, but in the same band. The wins on IFEval and BFCL trade against GSM8K and MMLU-Pro. For a decode-only engine still adding models, that’s the quality bar I’d want before claiming a speed crown.

Two honest caveats. First, these are RTX 5090 numbers — I haven’t re-run v0.3.7 on my RTX PRO 6000, so the headline table above and this section are different cards. Second, sparkinfer is still decode-only — no prefill/TTFT yet — so TRT-LLM remains the chat-latency pick on the PRO 6000. But for sustained decode on consumer Blackwell, v0.3.7 is the king. The eval pipeline backs the claim: dual-model scoring, same-box baselines against origin/main, and every run committed as raw log + result JSON with a per-run URL — a better bar than most “we’re faster” screenshots.

Memory is the other axis

Decode speed isn’t the only thing quantization buys. At BF16, both datacenter engines sit at ~91–93.5 GB of the 95 GB ceiling — Qwen3-30B-A3B at BF16 with a 32K KV cache is essentially the practical floor for this card; an 80 GB A100/H100 couldn’t hold it without quantizing or shortening context. Q4_K_M runs the same model in ~21 GB — meaning the GPU that barely fits one BF16 instance fits four Q4_K_M instances. Quantization is a density lever as much as a speed one.

128K context, cold starts, and the rest

A couple of practical findings that don’t fit the headline tables but decide real deployments:

• Extended context (>40K): only vLLM enabled 128K at the API level, via hf_overrides YaRN (factor 3.2 → ~131K window): TTFT 767 ms, decode 154 tok/s. TRT-LLM 1.x doesn’t expose rope_scaling in its LLM() API (you must bake YaRN into the checkpoint and rebuild). llama.cpp needs a GGUF with YaRN metadata pre-baked. If you need long context now with one flag, vLLM is the pick.
• Cold start: llama.cpp ~5 s (it just mmaps the weights), TRT-LLM ~90 s (FlashInfer JIT kernel compile), vLLM ~455 s — about 7 minutes the first time (torch.compile + a 419 s profiling/warmup pass), cached down to 0.78 s warm (a 535× speedup). For iterative experimentation, cold-start cost can dominate the throughput numbers you came for.

What I’d actually pick

The takeaway

The roofline decides, not the brand name on the engine. Prefill is compute-bound, decode is memory-bandwidth-bound, and a 4-bit quant trades a sliver of quality for ~4× less decode bandwidth and 4× the density — which is why a hobbyist-friendly engine beats two datacenter engines on the one metric (decode) that dominates long-form generation, while losing the one (TTFT) that dominates chat.

None of this is knowable from a spec sheet or a vendor benchmark. It came from running three engines in the same regime on my PRO 6000 — same GPU, same model, same 28K prompt — and reading the numbers honestly, including the parts that don’t flatter a tidy narrative. On consumer Blackwell, sparkinfer v0.3.7 is the decode king I haven’t re-benchmarked on my card yet. That’s the whole point of this blog: measure your actual workload on your actual hardware, because the answer flips depending on which half of the forward pass you’re standing in — and which GPU you’re standing on.