Tag: profiling

All the articles with the tag "profiling".

• TRT-LLM vs vLLM vs llama.cpp on Blackwell — sparkinfer v0.3.7 owns decode on the RTX 5090

  28 Jun, 2026

  sparkinfer v0.3.7: ~30% faster decode than llama.cpp on RTX 5090 at the same LLM quality — 2.5 MB native binary. I develop and maintain it. Plus TRT-LLM vs vLLM vs llama.cpp on PRO 6000.

• genie-ai-runtime v1.1 → v1.3.1: Gemma 4 on a Jetson, from llama.cpp parity to beating it at depth

  19 Jun, 2026

  From v1.1 to v1.3.1: Gemma 4 E2B on the Jetson Orin Nano — decode past llama.cpp via split-K attention plus a TTFT win, all driven by the profiler.

• Developing a custom Qwen2.5 72B inference engine from scratch — what is the real bottleneck?

  11 Jun, 2026

  Custom Qwen2.5-72B engine on 8×H200: the real bottleneck isn't the 72B matmuls — it's the 160 all-reduce barriers per token. H200 roofline math, barrier vs. bandwidth anatomy, and why the engine anti-scales at TP=8.

• Seven hard-won lessons from putting a zk-STARK prover on the Blackwell RTX PRO 6000 — 245 hand-written CUDA kernels

  20 Jan, 2026

  245 hand-written CUDA kernels on the RTX PRO 6000 Blackwell: seven performance-engineering lessons from GPU-accelerating a zk-STARK prover — profiling, memory, PTX, registers, and what Blackwell changed.