Tag: performance

All the articles with the tag "performance".

• TRT-LLM vs vLLM vs llama.cpp on Blackwell — sparkinfer v0.3.7 owns decode on the RTX 5090

  28 Jun, 2026

  sparkinfer v0.3.7: ~30% faster decode than llama.cpp on RTX 5090 at the same LLM quality — 2.5 MB native binary. I develop and maintain it. Plus TRT-LLM vs vLLM vs llama.cpp on PRO 6000.

• Training a DFlash drafter on a B200 — the real grind behind block-diffusion speculative decoding

  24 Jun, 2026

  I trained a DFlash drafter for a 72B on 2×B200 with the latest SGLang: the version-drift tax, the warm-start trap, accept 1.49→1.71, and what's next.

• genie-ai-runtime v1.1 → v1.3.1: Gemma 4 on a Jetson, from llama.cpp parity to beating it at depth

  19 Jun, 2026

  From v1.1 to v1.3.1: Gemma 4 E2B on the Jetson Orin Nano — decode past llama.cpp via split-K attention plus a TTFT win, all driven by the profiler.

• I implement the papers, I don't just read them — a speculative-decoding field journal

  19 Jun, 2026

  How I actually learn new inference tech: build it. Training EAGLE3 and DFlash drafters for a 72B, the traps reading never warns you about, and DFlash v2.

• Developing a custom Qwen2.5 72B inference engine from scratch — what is the real bottleneck?

  11 Jun, 2026

  Custom Qwen2.5-72B engine on 8×H200: the real bottleneck isn't the 72B matmuls — it's the 160 all-reduce barriers per token. H200 roofline math, barrier vs. bandwidth anatomy, and why the engine anti-scales at TP=8.

• genie-ai-runtime: a Jetson edge LLM inference engine that beats llama.cpp — designing for unified memory

  19 Apr, 2026

  A CUDA LLM engine built for Jetson's unified memory: ~2× llama.cpp short-prompt prefill on Qwen3-4B, decode ceiling math, and a transformer vs Mamba vs hybrid edge benchmark.