I keep a public repository that is, honestly, my real resume: ai-hardware-engineer-roadmap. It’s where I write down what I learn every day, and it states the goal in one line:
Master AI inference, AI agent harness systems, and hardware engineering — then design a physical AI chip.
It crossed 200 GitHub stars, which tells me I’m not the only one who wants to walk this path. This post is the thinking behind it — how I’ve come to frame “efficient AI,” and why that framing points at a single die at the end of the road.
Table of contents
Open Table of contents
Efficient AI has three layers
Whenever someone says “efficient AI,” they usually mean one slice of it. I’ve found it clearer to split the whole thing into three layers:
- AI algorithm — the model itself. New architectures, attention variants, quantization formats, sparsity, distillation. This layer decides how much work there is to do at all.
- AI system (MLSys) — the engine that runs the model. Kernels, schedulers, memory managers, KV cache, multi-GPU parallelism, the C++/CUDA that turns a model into tokens per second. This layer decides how efficiently the work gets done.
- AI application — the agent. Tools, planner/executor loops, sessions, memory, RAG, evals. This layer decides what the work is for.
The mistake is treating these as three separate study tracks. They’re a stack, and real efficiency only shows up when you co-design across them. A model architecture that ignores memory bandwidth is a slow kernel waiting to happen. A kernel that ignores the agent’s latency budget optimizes the wrong percentile. An agent that ignores inference cost is a demo, not a product. The interesting work lives at the seams.
The loop: what AI can do, where it runs, how to break its limits
Here’s how I actually move through those layers — as a loop, not a checklist:
- What can AI do? → learn the agent. Start at the application layer, because that’s the workload. Until you’ve built an agent harness — tool calls, multi-agent loops, the whole Claw War of frameworks — you don’t really know what the silicon underneath will be asked to do. The workload defines everything downstream.
- Where does AI run? → master the hardware. Then go down to where the tokens are actually made: Hopper and Blackwell in the datacenter, Jetson unified memory at the edge. You can’t optimize a system you don’t understand at the level of HBM bandwidth, NVLink barriers, SM occupancy, and the unified-memory budget on an 8 GB board.
- How do you break the hardware limit? → the algorithm layer. When the hardware says “no,” the answer often isn’t more hardware — it’s an algorithm that changes what the hardware is asked to do. Flash-offload decoding (DFlash-style) to run a model larger than VRAM, per-layer embeddings (PLE) to shrink the memory footprint, speculative decoding to cut the bandwidth bill. Each one bends a hard limit by attacking it from the math side.
- Then run it backwards → design the hardware for the algorithm. This is the part that closes the loop and points at the chip. Once you know the workload (agent), the machine (GPU), and the algorithmic tricks that beat the limits, you can ask the inverted question: what would the silicon look like if it were built for my proprietary algorithm, for one specific use case? Not a general-purpose GPU — a die shaped around exactly the inference I need to run.
That last step is co-design in the truest sense. The inference workload tells you what the silicon must accelerate; the agent harness tells you what the runtime must support; the hardware constraints tell you what power, memory, and I/O are actually real. The roadmap calls these the three pillars — inference engineering, agent harness systems, and physical hardware engineering — and the chip is the artifact where they converge.
Where I am right now
Two things, concretely.
Learning AI agents and NVIDIA GPU architecture in parallel — the top and bottom of the stack at the same time, because the agent tells me what to build and the GPU tells me what’s possible.
Building a custom AI inference engine that deliberately supports cutting-edge models and the optimization methods that matter: FlashInfer kernels, speculative decoding, EAGLE-3 draft heads, and more as they land. The point isn’t to re-implement vLLM. It’s to learn, in my own hands, how a clever algorithm and tight C++/CUDA combine to make AI genuinely efficient and useful — the exact seam between layers 1 and 2.
This isn’t abstract. It’s the same thread running through everything on this blog:
- Inference systems — a Jetson edge engine that beats llama.cpp, a custom 72B engine on 8×Hopper, and 245 CUDA kernels for a zk-STARK prover on Blackwell.
- Agent harnesses — genie-claw on-device and maintaining OpenClaw.
- Hardware — the FPGA→GPU build log that taught me timing closure, SLR crossings, and why I’d reach for a GPU over an FPGA for a fast-moving target.
Each one is a different face of the same loop.
The destination: an agentic AI inference chip
The honest version of the five-year plan: build my own agentic AI inference chip. A single die that runs production inference, hosts a real agent runtime, and talks to the physical world — a Jetson-class AI brain fused with an ESP32-class wireless stack, shaped around one specific class of agentic workload instead of trying to be everything to everyone.
That’s an enormous goal, and I’m under no illusion about the distance. But the path decomposes cleanly: master the inference, master the agent harness, master the hardware — and let each one inform the design of the next. The roadmap repo is the plan. This blog is the lab notebook that fills it in, one measured experiment at a time.
If you’re walking a version of this path too — or you’re building the silicon, the runtime, or the kernels that live on it — I’d like to hear from you.