I develop sparkinfer — the Blackwell-native decode engine from the 4-engine post. sparkinfer-k3 is the fork of it aimed at one specific, brutal target: Kimi K3, 2.8 trillion parameters, hybrid KDA + MLA attention, 896 routed experts — the largest open-weight model that exists, on a single rentable 8× H200 box. Four releases in under a week, v0.1.0 through v0.2.2, took it from a 1 tok/s standing start to 3.26× llama.cpp on decode. This post is what that climb actually looked like, the bugs it forced into the open, and what “moving fast” means when every PR has to prove itself on a pinned node before it’s allowed to count.
Table of contents
Open Table of contents
Why sparkinfer-k3 needs to exist
sparkinfer proper is explicit about its scope, right in its own README: “Optimized for every Blackwell GPU” — sm_120/sm_121 only, a lineup that runs RTX 5090 → RTX PRO 6000 → RTX Spark/DGX Spark, and a deliberately short “frontier models” list (Qwen3.6-35B-A3B, Qwythos 9B, MiniMax M3 next). That’s the right scope for its mission — consumer and workstation Blackwell, cost-effective tokens per dollar at the edge — and it means nothing on that roadmap is within an order of magnitude of 2.8 trillion parameters, and none of it targets a datacenter Hopper card like the H200.
Kimi K3 misses on both axes sparkinfer optimizes for, at once. Architecture: KDA + MLA hybrid attention, a latent MoE over 896 experts, a situ/cross-layer-residual design, and a tiktoken BPE vocabulary instead of a standard tokenizer.json — no engine’s kernels, shard policy, or tokenizer path were written for this shape, sparkinfer’s included. Scale: 553 GiB of weights at the lowest usable quant, across 93 layers — no card in sparkinfer’s own target list holds that, and MoE sharding built for a handful of dozens of experts doesn’t generalize to 896 of them. So sparkinfer-k3 isn’t a port, it’s a fork with its own loader, its own ShardPolicy::ExpertsOnly, and a tokenizer path built around the GGUF’s own tiktoken vocab rather than one that would silently produce different ids.
The core trick that makes 896 experts tractable per-GPU: shard the expert stack across ranks instead of replicating it. In the v0.1.0 tensor-parallel bench, rank-0 residency drops 94.08 GiB (tp=1) → 24.25 GiB (tp=8) — a 3.9× per-rank reduction, which is the difference between “fits on an H200” and “doesn’t.”
The reasons it’s worth building at all track the K3 repo’s own pitch: running the single largest open-weight model locally — no API, no per-token bill, nothing leaving the building; for enterprises (legal, medical, defense, finance) whose entire requirement is that the data never leaves; for anyone building a private AI product who wants frontier-class weights they actually control; and for inference engineers who want every kernel and shard decision readable in one repo instead of black-boxed behind someone else’s endpoint. And it sits directly on sparkinfer’s own published roadmap: Milestone 2 there is “Trustable AI on confidential compute” — TDX + NVIDIA CC attestation, sealed execution, models sourced from SparkDistill. Running the largest open-weight frontier model at all — fast, correctly, provably — is the prerequisite that has to exist before “run it privately” is even a question worth asking. sparkinfer-k3 is that prerequisite, at the datacenter end of the same roadmap sparkinfer is building at the consumer end.
v0.1.0 — 1.01 → 18.87 tok/s in under two days
The 128k decode track opened on Aug 1 at 1.01 tok/s — 6% of the llama.cpp reference. What happened next is the whole story of this project in one table:
UTC merged PR 128k decode vs llama.cpp (16.70 tok/s)
08-01 track opens 1.01 tok/s 6%
08-01 17:44 #49 4.53 tok/s 27%
08-01 23:44 #57 9.04 tok/s 54%
08-02 01:34 #59 17.46 tok/s 105% ◄── crossed llama.cpp
08-02 03:49 #67 18.14 tok/s 109%
08-02 ~05:00 #73 18.87 tok/s 113%18.7× in under two days, crossing the reference engine six hours after the track opened, with top-1 accuracy holding at 100% and mean KL divergence halving along the way — the speed didn’t come at the cost of correctness. Four contributors carried the climb: @inference2026 split MLA decode over context (the 1.01 → 4.53 leg), @widecloud batched MLA heads per block (4.53 → 9.04), @FranDev132 head-sharded attention with a split-cap budget (9.04 → 16.06), and @flashatten’s f32 peer one-shot all-reduce landed the actual crossing.
None of that is claimed on anyone’s word. Every run is timed from outside the binary, gated by a two-sided wall-clock bound (a benchmark that skips work is refused, same as one that overclaims), and sealed into an Ed25519-signed receipt in sparkinfer-k3-log — an append-only ledger, verifiable without a GPU. If that sounds familiar, it’s the same discipline as the attestation work I shipped on SparkDistill: don’t ask anyone to trust the number, make the number check itself.
v0.2.0 — 8 merged PRs, one bug worth remembering
One day later: 18.14 → 41.39 tok/s, 2.24× llama.cpp, an eight-PR ladder from value-tiled KDA decode through 2-D MoE sharding to a warp-budget projection tier that closed the day at 41.39.
The instructive part isn’t the ladder — it’s what almost broke it. PR #81 (“one quantise per activation”) measured a gain, got reverted (#84) when the number looked like it came from a stale build directory, then got restored (#94) after twelve clean builds confirmed a genuine +4.1%. That’s the correct failure mode for a competitive eval system: revert on doubt, but don’t let a false-negative kill real work either — verify the doubt itself before you let it stand.
A quieter bug taught me more. Tier credit was capped at twice the measured speedup — a rule that’s invisible while the frontier sits below the reference and becomes load-bearing the moment it doesn’t. Once sparkinfer passed 2× llama.cpp, an XL-tier PR was only getting credited for 9% of its real 18% gain. The bug was dormant for the entire early climb and only started lying once the project succeeded. That’s a class of bug worth watching for generally: constraints derived from “where we are now” quietly become wrong the moment progress moves past the assumption baked into them.
v0.2.1 — fixing the benchmark, not just the kernel
41.39 → 56.82 tok/s, 3.08× llama.cpp, on three merged PRs — but the changelog here is really about the eval itself getting harder to fool, which mattered more than any single kernel:
- Parity had been graded on a single 4-token prompt — a KV cache essentially empty, deciding whether a change held accuracy at the 131,072-token context the engine is actually scored on. It now sweeps 4 → 4096 tokens as nested prefixes and takes the worst depth. A shallow check that happens to pass isn’t the same as a check that can’t be gamed at depth.
- The tier-credit cap from v0.2.0 got fixed at the root: capped at the measured speedup, not twice it. Eight prior runs were re-scored from the sealed receipts once the fix landed — the receipts existed precisely so that a rule change could be re-applied to history instead of trusted retroactively.
- The wall-clock guard’s margin was still sized for a 2× frontier on a system now running past 3×; one round produced three mutually exclusive failures of the same check on code that hadn’t changed. The guard has to scale with the thing it’s guarding.
v0.2.2 — knowing when to move the goalposts
By v0.2.1, decode was 3.08× ahead of llama.cpp — comfortably won. Prefill was 3.57× behind. Scoring decode further would have kept rewarding the metric that had already stopped being the bottleneck. v0.2.2 made the call explicitly: prefill becomes the scored metric; decode becomes a regression guard that refuses any PR dropping more than 1% below the frontier. Two days later: prefill 40.35 → 99.68 tok/s (2.47×), decode untouched at 60.17 tok/s (3.26×).
The headline change was batched prompt ingestion (#148): prefill had been one forward pass per token through all 93 layers. Moving the layer loop outside the token loop means a chunk of tokens shares a weight-tile read across the chunk instead of re-reading it per token — output bit-identical to the old per-token walk at every chunk width. Simple idea, real payoff.
That PR also surfaced the two best bugs in the whole series, and both are worth keeping in a mental “bug shapes” catalog:
A swapped argument pair that was invisible below 12 layers. attn_res_mix_f32 takes (n_rows, act_row_stride, bank_row_stride, …); all three call sites passed (n_rows, bank_row, H) — arguments two and three swapped. It never showed up because bank_row happens to equal res_bank_row_elems * ceil(n_layers / 12), so at ≤12 layers the two swapped values are numerically equal and each wrong argument lands on the right number by coincidence. It took a model deep enough to bank twice to expose it. Correctness bugs can hide behind a coincidence that only breaks at scale — the fact that a kernel passes at your test model’s depth proves nothing about a bigger one.
A DEFAULT OFF flag that was actually on. The check was !(e && e[0] == '0') — read as “off unless explicitly set to 0” instead of “on only if explicitly set.” An unset environment variable made the polarity invert, and every chunk of two or more tokens died at launch. The comment said DEFAULT OFF in two places; the code did the opposite. When a flag’s comment and its logic disagree, trust the logic and go looking for why — the comment is the one that’s usually stale.
There was also a stale-baseline bug worth a line: the prefill frontier had been pinned at 40.35 from a commit that predated three already-merged decode PRs, while main genuinely measured 53.02. Three contributors optimized against the wrong bar, measured honestly, and still landed under it through no fault of their own. reconcile_lock now flags a pin stale once it drifts more than 10% — the fix isn’t “measure more carefully,” it’s “make the reference incapable of silently going out of date.”
What I learned from it
A few things that generalize past this one repo:
- A benchmark you can’t fool is worth more than a kernel that’s fast. Most of what actually shipped across these four releases was eval-hardening — parity depth, tier-credit math, wall-clock margins, a pin that flags its own staleness — not raw kernel work. The kernels were only trustworthy because the harness around them kept getting harder to game.
- Constraints derived from “current state” expire. The 2×-speedup tier cap and the 2×-scaled wall-clock margin were both correct when written and both silently wrong once the project succeeded past the assumption baked into them. Any threshold set relative to where a system is now needs a plan for what happens when it isn’t there anymore.
- Scale exposes bugs that small tests hide by coincidence. The swapped-argument bug and the polarity-inverted flag both passed at the model sizes and chunk widths people had actually been testing. Neither was subtle in the code — they were subtle in the fact that nothing exercised the input range where they mattered.
- Score the actual bottleneck, not the metric you’re already winning. Once decode passed llama.cpp by 3×, continuing to optimize decode would have been optimizing a solved problem. Moving the scored metric to prefill — the thing still losing — is the same discipline as profiling before you optimize, applied to an entire project’s roadmap instead of one kernel.
Moving fast on Gittensor
SN74 on Gittensor is the mechanism that made this pace possible, not just the backdrop for it. Every PR builds against main on the same pinned 8× H200 node the frontier is measured on; correctness (top-1 ≥ 0.95, KL ≤ 0.05 vs llama.cpp) gates before any speed tier is even considered; a regression cannot merge, full stop; and the frontier number that everyone is racing against updates the moment a PR clears the bar — not on a release cadence. That loop is why a track that opened at 1.01 tok/s could cross the reference engine the same day, and why six named contributors (@inference2026, @widecloud, @FranDev132, @flashatten, @JSONbored, @marktech0813) could each land one leg of the climb without stepping on each other — the eval, not a maintainer’s judgment call, decides what counts.
That’s the throughline across everything I’ve built on Gittensor: SparkDistill proves training claims cryptographically instead of taking a miner’s word; sparkinfer-k3 proves speed claims the same way — sealed receipts, external wall-clock bounds, an eval that gates before a number is allowed to count. Fast and honest aren’t in tension here. The honesty is what makes the speed real instead of a screenshot.