Tag: ai-inference
All the articles with the tag "ai-inference".
-
How we got Qwen3.8-27B to 92.9 tok/s and a 360K window on one RTX 5090
The engineering behind our NVFP4 build of Qwen3.8-27B: why decode is a bandwidth problem, why lm_head was the tensor that mattered, and the exact environment it was built and measured in.
-
830K downloads later: what quantizing Qwen3.8-27B to NVFP4 for one RTX 5090 actually taught me
Our NVFP4 build of Qwen3.8-27B passed 830K monthly downloads. Then NVIDIA and two others published their own. I benchmarked all three on one 5090 — quality was a tie, and what separated them was precision allocation, a drafter trained against the checkpoint, and the runtime underneath.
-
18.7× in two days: what sparkinfer-k3 taught me about shipping fast on Gittensor
sparkinfer-k3 took Kimi K3 (2.8T) from 1 tok/s to 3× llama.cpp on 8× H200 in days. The climb, its bugs, and what it taught me about racing on Gittensor's SN74.
-
TRT-LLM vs vLLM vs llama.cpp on Blackwell — sparkinfer v0.3.7 owns decode on the RTX 5090
sparkinfer v0.3.7: ~30% faster decode than llama.cpp on RTX 5090 at the same LLM quality — 2.5 MB native binary. I develop and maintain it. Plus TRT-LLM vs vLLM vs llama.cpp on PRO 6000.
-
The confidential API: sending a prompt to a cloud GPU the provider can't read
HTTPS protects the wire, not the server. A confidential API binds an attested TEE to an ECDH session key — so prompts stay encrypted even from the provider.
-
Trust before secrets: confidential AI inference with CPU TEEs and GPU Confidential Computing
Renting a GPU means trusting someone else's host. Confidential computing flips it: attest the CPU TEE and GPU, then release API keys only to a verified CVM.