I was never an “AI agent person.” I followed the news, watched the YouTube videos, and mostly shrugged. The eras looked clear enough from the outside: 2023 was the GPT era, OpenAI’s moment; 2024–25 was the reasoning era, with DeepSeek making everyone rethink what a model could do on its own; 2025 was when “agentic AI” became the phrase on every slide. I noted it and moved on.
2026 felt different. This year the AI agent stopped being a slide and became the product — and it pulled my whole field, inference, along with it.
Table of contents
Open Table of contents
The OpenClaw moment
It clicked for me watching Jensen Huang at NVIDIA GTC 2026. NVIDIA’s CEO stood on stage and called OpenClaw — a viral open-source AI agent platform — “the new computer,” and told the room that every company now needs an “OpenClaw strategy.” Software, he argued, is shifting from chatbots you talk to into autonomous agents that act on your behalf. NVIDIA even shipped an enterprise stack for it, NemoClaw.
Then I looked at the star history, and it didn’t look like a normal project — it went vertical. OpenClaw blew past React to become the most-starred repository on GitHub, thousands of stars a day. Within weeks a whole menagerie appeared around it: hermes-agent, zeroclaw, picoclaw, nemoclaw, nanoclaw, a wave of security-hardened “claws,” each crossing tens of thousands of stars in a month. It felt like React arriving in web development — except this wasn’t a framework war inside one field. It was a worldwide tsunami.
The Claw War
OpenClaw going viral didn’t end the agent race. It started one.
Every framework in the menagerie was betting on a different theory of what an AI agent actually needs:
- hermes-agent — the “do it right” camp. Minimum 64k context, full plan/observe/reflect/act loop, persistent memory, multi-agent coordination out of the box. Heavy, but it works for real enterprise workflows. The argument: 16k context isn’t enough for anything useful; you need room for the plan, the tool outputs, the prior observations, and the next step all at once.
- zeroclaw — the “do it fast” camp. Stripped the memory system, trimmed the context, made the tool-call loop as thin as possible. First-generation agents needed complexity; second-generation agents need speed. The claim: most agent tasks complete in 3–5 tool calls; a smaller loop means lower latency per step.
- picoclaw / nanoclaw — the “do it anywhere” camp. Targeting edge and mobile — phones, Jetson boards, embedded devices. The bet: the agent future isn’t cloud-only; it’s every device running a private, always-on agent. Memory budgets in the hundreds of MB, context budgets in the thousands of tokens, accuracy from grounding rather than scale.
- nemoclaw (NVIDIA) — the “do it on our silicon” camp. Enterprise stack, proprietary optimizations for NVIDIA hardware, tight integration with TensorRT-LLM. NemoClaw is less a framework and more a statement: NVIDIA wants to own the inference layer underneath every agent, and the agent framework is the distribution mechanism.
- Security claws — the “do it safely” camp. Sandboxed execution, audit trails, tool-call rate limits, PII redaction, egress controls. As agents started acting on your behalf — writing files, sending emails, calling APIs — the attack surface exploded, and a wave of security-hardened forks appeared from practitioners who’d already been burned.
Each was a different answer to the same question: what does it cost to run an agent, and what do you have to give up to pay that cost? The Claw War wasn’t about which framework had the best README. It was about which framework’s cost model survived contact with production.
What actually changed
Agents changed how we use models. A chatbot answers once; an agent plans, calls tools, spawns sub-agents, reads your files, and keeps going. That single shift detonated token usage — one agent task can burn what a hundred ChatGPT prompts used to.
Concretely:
- OpenClaw’s minimum context: 16k tokens
- hermes-agent’s minimum: 64k tokens
- A reasoning trace on a 5-step task: easily 10k–50k tokens
- GPT-4-class prompt + tool outputs + plan + history: routinely 32k–128k tokens
One engineer’s agent loop is more tokens per second than a mid-sized consumer app was in 2023. The scale jumped by an order of magnitude overnight — and it jumped at the exact moment models started reasoning more, not less. DeepSeek’s reasoning models, then the reasoning variants of Qwen and Llama, then the o-series successors — all of them producing longer internal traces, more tokens per answer, more pressure on the inference layer.
And the loop reinforces itself: at Computex, in the Jetson talk, agents were already helping design the next chips. AI helping design the silicon that runs AI, which makes tokens faster and cheaper per watt, which makes more agents viable. A flywheel.
The inference battlefield: TTFT, throughput, concurrency
This is the part I care about most, because it is my field. When agents became the workload, the rules of inference hardened into three simultaneously:
| Metric | Why it matters for agents |
|---|---|
| TTFT (time to first token) | Agents are interactive and chained — latency compounds across every step in the loop |
| TPS (tokens per second, throughput) | Reasoning traces and tool-use outputs are long; slow generation means slow agents |
| Concurrency (requests/second, batch) | One “user” is now dozens of concurrent agent calls; a single enterprise workflow fans out massively |
Fast first token, fast generation, and many at once — all three. Not optional anymore. The rule.
At TP=8 on a Qwen2.5-72B serving cluster, the numbers make this concrete: vLLM scales decode throughput from 44 → 92 tok/s as you go from 2 to 8 GPUs, because it’s bandwidth-bound and more GPUs means more HBM bandwidth. An engine that handles the synchronization wrong can regress on the same hardware — I measured 44 → 38 tok/s before fixing the all-reduce barrier path. The difference between 92 and 38 tok/s isn’t a 2.4× difference in user experience; at agent scale — where one user’s workflow fans into dozens of parallel LLM calls — it’s the difference between a working product and an unusable one.
The fastest token — what the speed race looked like
By mid-2026, “fastest LLM inference” had become a legitimate competitive claim, not just a benchmark exercise. Three fronts:
1. The serving engine race. vLLM, SGLang, TensorRT-LLM, and a wave of funded startups were publishing competing throughput numbers weekly. The game shifted from “serving a chatbot” to “serving 10,000 concurrent agent sub-tasks.” vLLM’s continuous batching and paged attention were the 2023–24 baseline; by 2026 the frontier was speculative decoding at scale (verify multiple draft tokens in one weight-read), disaggregated prefill/decode (separate hardware for the two phases), and MoE routing optimizations (only activate the relevant experts per token, so each token burns a fraction of the total parameter count).
2. The chip race. The H200 SXM’s 3.35 TB/s HBM3e bandwidth raised the hardware ceiling. Blackwell’s B200 added FP4 inference and NVLink 5.0’s 900 GB/s per-GPU interconnect. Every architecture announcement was immediately translated into “how does this change the inference math?” — because the bottleneck always moves: more compute → more bandwidth-bound; more bandwidth → more barrier-bound on the collectives; fix the collectives → more compute-bound again. The roofline doesn’t get simpler, it just shifts.
3. The model architecture race. When context windows got long and KV caches got expensive, the architecture response arrived fast. Mamba and Mamba-2 demonstrated that SSM-based models could match transformer quality at a fraction of the KV memory cost. Falcon-H1 showed that parallel attention + Mamba-2 hybrid blocks could run faster than a pure transformer at long context while keeping KV growth sub-linear. Diffusion-based LMs out of Google proposed generating multiple tokens simultaneously instead of one at a time — a fundamentally different attack on the token rate problem. The model zoo is not converging; it’s diversifying, because each architecture is a different trade on the speed/memory/quality Pareto frontier.
The number I keep coming back to: on a Jetson Orin Nano 8 GB (the device in my living room), a 4B transformer model running llama.cpp generates ~18 tok/s on a short prompt. My custom kernel on the same hardware generates ~38 tok/s. At 18 tok/s, a 3-sentence home command takes ~4 seconds to respond. At 38 tok/s, it takes ~2 seconds. At the target of ~60+ tok/s (speculative decode, domain-tuned head), it approaches human speech pace. The “fastest token” competition isn’t abstract. On an edge box, every 10 tok/s is a second of wall-clock latency the user has to wait through.
The deeper competition: who owns the inference layer
The cleanest signal that 2026 was different wasn’t any single framework or benchmark. It was the strategy moves:
- Qwen — long the open-weight champion — shipped its first closed-weight flagship and pivoted to agent-first models. Open the agent layer, keep the model-and-silicon stack proprietary. The reasoning: if agents are the product, whoever the agent calls is the strategic position, and the model at the center of every agent call is the monetization point.
- NVIDIA shipped NemoClaw alongside NIM (NVIDIA Inference Microservices) — a bundled agent framework + optimized inference container. The message: NVIDIA isn’t just the GPU vendor anymore. It wants to own the software stack from kernel to agent.
- Anthropic, Google, OpenAI each launched or deepened their own agent APIs — hooks that make it easy to build on their models as the agent’s reasoning core. The inference speed of those APIs became a product differentiator, not just an engineering metric.
Every major player was trying to answer the same question: if agents are the new compute primitive, who controls the inference layer underneath them? The Claw War was the application-layer expression of that question. The inference arms race was the infrastructure layer.
Why I jumped in
This is what pulled me in. I started contributing to OpenClaw out of pure interest — not for a salary, just because it was the most alive thing in the field. That turned into something real: I’m now a maintainer of OpenClaw (my merged PRs).
It also sharpened exactly what I build:
- A custom 72B inference engine — pure C++/CUDA, no Python, targeting Qwen2.5-72B on 8×H200 to beat vLLM. With custom communication, custom kernels, a hand-rolled barrier-free all-reduce, I beat vLLM on a single request at 32k context — on both TTFT and decode throughput at TP=4. I have not beaten it on concurrency yet, and that is the genuinely hard, beautiful part of vLLM. The engineers behind it are excellent; closing that gap is the work. (Details in the post)
- genie-ai-runtime — a Jetson-native CUDA engine that runs Qwen3-4B at ~38 tok/s short-prompt prefill vs llama.cpp’s ~18 tok/s, with a unified-memory-aware budget system and a prefix KV cache that makes the system prompt effectively free. (Details here)
- genie-claw — my own agent, on-device for 100% privacy, accurate inside 4096 tokens because it’s grounded in a deterministic device catalog instead of relying on a giant context window. (Details here)
The thread is the same in all three: the inference layer is the thing that determines whether the agent works or doesn’t. Fast tokens aren’t a benchmark metric. They’re the product.
Where this goes
The future here is more competition, not less. Claw wars between agent frameworks; efficiency wars between serving engines; architecture wars between models; and hardware wars between chip vendors. The competitive moat is shifting: in 2023 it was the model. In 2025 it was the reasoning trace. In 2026 it is the inference infrastructure that runs the agent — latency per step, tokens per second, cost per million tokens, memory per context.
That’s exactly why I want to be a top 0.1% AI inference optimization engineer — toward 2027, 2030, 2040. Every layer of the stack is in motion, the demand curve only steepens, and it will reward the people who can squeeze the hardware harder than anyone else. The seat I’m working toward is at the bottom of that stack, where tokens get made.