Tag: ai-inference
All the articles with the tag "ai-inference".
-
TRT-LLM vs vLLM vs llama.cpp on Blackwell — sparkinfer v0.3.7 owns decode on the RTX 5090
sparkinfer v0.3.7: ~30% faster decode than llama.cpp on RTX 5090 at the same LLM quality — 2.5 MB native binary. I develop and maintain it. Plus TRT-LLM vs vLLM vs llama.cpp on PRO 6000.
-
The confidential API: sending a prompt to a cloud GPU the provider can't read
HTTPS protects the wire, not the server. A confidential API binds an attested TEE to an ECDH session key — so prompts stay encrypted even from the provider.
-
Trust before secrets: confidential AI inference with CPU TEEs and GPU Confidential Computing
Renting a GPU means trusting someone else's host. Confidential computing flips it: attest the CPU TEE and GPU, then release API keys only to a verified CVM.
-
Training a DFlash drafter on a B200 — the real grind behind block-diffusion speculative decoding
I trained a DFlash drafter for a 72B on 2×B200 with the latest SGLang: the version-drift tax, the warm-start trap, accept 1.49→1.71, and what's next.
-
genie-ai-runtime v1.1 → v1.3.1: Gemma 4 on a Jetson, from llama.cpp parity to beating it at depth
From v1.1 to v1.3.1: Gemma 4 E2B on the Jetson Orin Nano — decode past llama.cpp via split-K attention plus a TTFT win, all driven by the profiler.
-
I implement the papers, I don't just read them — a speculative-decoding field journal
How I actually learn new inference tech: build it. Training EAGLE3 and DFlash drafters for a 72B, the traps reading never warns you about, and DFlash v2.