Tag: ai-inference

All the articles with the tag "ai-inference".

• 18.7× in two days: what sparkinfer-k3 taught me about shipping fast on Gittensor

  6 Aug, 2026

  sparkinfer-k3 took Kimi K3 (2.8T) from 1 tok/s to 3× llama.cpp on 8× H200 in days. The climb, its bugs, and what it taught me about racing on Gittensor's SN74.

• TRT-LLM vs vLLM vs llama.cpp on Blackwell — sparkinfer v0.3.7 owns decode on the RTX 5090

  28 Jun, 2026

  sparkinfer v0.3.7: ~30% faster decode than llama.cpp on RTX 5090 at the same LLM quality — 2.5 MB native binary. I develop and maintain it. Plus TRT-LLM vs vLLM vs llama.cpp on PRO 6000.

• The confidential API: sending a prompt to a cloud GPU the provider can't read

  25 Jun, 2026

  HTTPS protects the wire, not the server. A confidential API binds an attested TEE to an ECDH session key — so prompts stay encrypted even from the provider.

• Trust before secrets: confidential AI inference with CPU TEEs and GPU Confidential Computing

  25 Jun, 2026

  Renting a GPU means trusting someone else's host. Confidential computing flips it: attest the CPU TEE and GPU, then release API keys only to a verified CVM.

• Training a DFlash drafter on a B200 — the real grind behind block-diffusion speculative decoding

  24 Jun, 2026

  I trained a DFlash drafter for a 72B on 2×B200 with the latest SGLang: the version-drift tax, the warm-start trap, accept 1.49→1.71, and what's next.

• genie-ai-runtime v1.1 → v1.3.1: Gemma 4 on a Jetson, from llama.cpp parity to beating it at depth

  19 Jun, 2026

  From v1.1 to v1.3.1: Gemma 4 E2B on the Jetson Orin Nano — decode past llama.cpp via split-K attention plus a TTFT win, all driven by the profiler.