Tag: ai-inference
All the articles with the tag "ai-inference".
-
The three layers of efficient AI — my roadmap from inference kernels to a physical AI chip
Efficient AI splits into algorithm, system, and application — the co-design loop across all three, and why my roadmap ends in a custom AI inference chip.
-
Developing a custom Qwen2.5 72B inference engine from scratch — what is the real bottleneck?
Custom Qwen2.5-72B engine on 8×H200: the real bottleneck isn't the 72B matmuls — it's the 160 all-reduce barriers per token. H200 roofline math, barrier vs. bandwidth anatomy, and why the engine anti-scales at TP=8.
-
Deploying VLA models on Jetson — what fits, what doesn't, and what TensorRT buys you
VLA deployment on Jetson Orin — what models fit in 8–64 GB unified memory, ORT-TRT EP 5.55×, SnapFlow distillation to get pi0.5 down to 6.5 GB, and the engineering traps that bite every fresh Jetson setup.
-
genie-ai-runtime: a Jetson edge LLM inference engine that beats llama.cpp — designing for unified memory
A CUDA LLM engine built for Jetson's unified memory: ~2× llama.cpp short-prompt prefill on Qwen3-4B, decode ceiling math, and a transformer vs Mamba vs hybrid edge benchmark.
-
genie-claw: building a private on-device AI agent for Jetson — what is limited context size, and why it's the real engineering problem
On-device private home agent on Jetson Orin Nano 8 GB — what context size actually means, the KV cache memory math, 4096-token budget engineering, and BFCL accuracy from grounding.
-
AI agents vs. AI Inference: how they evolved and competed in 2026 — the Claw War, the fastest token, and what the race really is
2026: the OpenClaw wave, the Claw War between agent frameworks, and the inference arms race underneath — TTFT, throughput, and why fast tokens became the real competitive moat.