genie-claw: building a private on-device AI agent for Jetson — what is limited context size, and why it's the real engineering problem

I built genie-claw because I couldn’t stop thinking about a question I asked myself while contributing to OpenClaw: what if this runs entirely on your own hardware, without sending a single byte to the cloud?

The question sounds simple. The engineering answer is dominated by one number: context size. Not the model architecture, not the inference speed, not even the hardware. The binding constraint for an on-device AI agent is how much text the model can hold in its working memory at once — and on an 8 GB board, that number is brutally small compared to what the major agent frameworks assume.

Table of contents

The smart home that isn’t

Home Assistant is excellent software. It connects everything, runs locally, and has thousands of integrations. But it isn’t a smart home — it’s a remote control. You press a button, a light turns on. You write an automation, it fires on a schedule. There is no reasoning, no memory, no agent acting on your behalf. As of 2026, Home Assistant has no MCP server — no interface for a language model to call its tools directly. It’s a control bus waiting for an AI brain.

The “smart home” phrase has always been marketing. I wanted to build the real thing: an agent that lives in your house, knows your preferences, controls your devices, hears you speak, and replies — entirely on your hardware, never touching a cloud API.

Why privacy matters more at home

The privacy argument isn’t abstract when it’s your home. Your home is where your family is. Every spoken command is a data point about when you wake up, where you are, what you’re doing, who you’re with. I’m not comfortable streaming that to any provider’s logging pipeline, regardless of their privacy policy. The edge constraint isn’t a limitation to engineer around — it’s the product requirement.

The hardware choice followed: the Jetson Orin Nano Super 8 GB is the smallest NVIDIA board with CUDA, and CUDA is what the inference stack needs. 8 GB unified memory, SM 8.7, CUDA 12.6. That’s the first genie-claw target.

What is context size, exactly?

Before getting to the constraint, it’s worth being precise about what “context size” actually means — because the phrase is used loosely and the technical meaning matters for understanding why it’s expensive.

Tokens, not words. A language model doesn’t read text — it reads tokens, subword pieces produced by a tokenizer. A token is roughly 0.75 words on average in English (so 4096 tokens ≈ 3000 words, or about 5–6 pages of text). But that average hides a lot: common short words are one token each, rare words and proper nouns can be 3–5 tokens, and code or structured text (JSON tool manifests, device catalogs) is often denser than prose. A genie-claw tool manifest listing 40 home devices with their rooms and action schemas costs around 300–400 tokens — not “300–400 words” but 300–400 model input positions.

The context window as working memory. The context window is literally everything the model can “see” when it generates the next token: the system prompt, the conversation history, any injected memory, the tool definitions, and the current user turn. Nothing outside the window exists for the model — it can’t retrieve memories from outside it, it can’t refer back to turns that fell off the edge. The context window is the model’s working memory, and it’s fixed in size at inference time by the configuration you hand the runtime.

The KV cache: why context costs memory. During inference, the transformer maintains a KV (Key-Value) cache — two tensors per attention layer that store the intermediate activations for every token in the context so far. Each new generated token can attend back to all prior tokens without recomputing them. This is efficient, but it means the KV cache grows linearly with sequence length.

The memory cost for a typical 4B model (36 layers, 8 KV heads with GQA, head dimension 128, FP16):

KV cache size = 2 (K+V) × seq_len × layers × kv_heads × head_dim × 2 bytes (FP16)
             = 2 × seq_len × 36 × 8 × 128 × 2
             = seq_len × 147,456 bytes
             ≈ seq_len × 144 KB per token

What that works out to in practice:

These numbers are for the KV cache alone — before model weights, before the OS, before STT, TTS, or the agent process. The weights for a 4B model at Q4_K_M quantization are roughly 2.3 GB. On a Jetson Orin Nano with 8 GB unified memory, the full memory budget looks like this:

At 4096 tokens (~576 MB KV), you have ~2.9 GB margin — tight but manageable. At 16k tokens (~2.3 GB KV), you’re at ~1.15 GB remaining — a small OOM risk under any allocation spike. At 32k tokens (~4.6 GB KV), you’re over budget by more than 1 GB before the first token is generated.

The prefill cost on top. KV cache is a static memory cost. There’s also a compute cost: before the model can generate the first output token, it must process every token in the input — the system prompt, the memory, the history, the current turn — in a single forward pass called prefill. Prefill time scales roughly linearly with input length. On Orin Nano, processing a 4096-token context takes ~1–2 seconds. Processing a 16k context takes ~6–8 seconds — that’s the dead time between the user finishing speaking and the first word of the response.

This is what “limited context size” means in practice: memory and latency both scale with context length, and on an 8 GB board, both run out faster than you’d like.

The OpenClaw observation

I was watching the OpenClaw wave from the inside — contributing PRs, watching the star charts go vertical, watching Jensen call it “the new computer.” I understood the architecture. And I understood why it didn’t fit on a Jetson.

OpenClaw’s minimum viable context is ~16k tokens. It carries a large system prompt, persistent tool manifests, multi-turn history, and memory injection — all necessary for a general-purpose agent that can maintain coherent plans across many steps. The hermes-agent framework that grew around it starts at ~64k minimum — enough context to hold the conversation, the plan, the current observations, the prior tool results, all at once. OpenClaw itself needs around 1 GB of process memory just to run; the KV cache for its minimum 16k context adds another ~2.3 GB on top.

On an Orin Nano with 8 GB unified memory split between OS, audio stack, STT, TTS, model weights, and the KV cache — 16k context is an OOM waiting to happen. At 16k, the KV cache alone consumes the entire remaining headroom after fixed overheads. The architecture assumes abundant context. I couldn’t shrink OpenClaw; I had to rethink the architecture from scratch.

The 4096-token wall

The memory budget forced a number: 4096 tokens total — system prompt, memory injection, conversation history, tool manifest, and the current turn. All of it. The agent enforces this at boot via validate_limited_context_agent() and surfaces pass/fail in the /api/health endpoint. If the configured budget doesn’t fit within the hardware limit, the process refuses to start.

My first attempt failed. The early versions were brittle: context overflows pushed history off the window, important memory got dropped mid-conversation, multi-step tool calls fell apart when the early turns scrolled out of the window. It felt more like a toy than a home assistant. I genuinely wasn’t sure 4096 was enough to build something useful.

But the constraint also forced something good: I had to be precise about what the window contains, and disciplined about every token in it.

The 4096-token breakdown in genie-claw today:

Why the memory budget is hard-capped at 700 tokens. Without a cap, build_memory_context() would assemble every matching memory line and potentially consume the entire window with identity and preference records — leaving no room for the actual conversation. The 700-token cap enforces a strict priority order: identity facts first (who you are, your family), relationship context second, query-relevant memories third, preference records last. If the total would exceed 700 tokens, lower-priority lines are dropped. This is not a soft guideline — it’s enforced at runtime on every turn.

The 20-turn history story. I started max_history_turns = 20, the intuitive default for “keep lots of history.” On Orin Nano it caused a ~50 s LLM stall — not from slow inference, but from slow prefill. 20 turns of conversational history is roughly 2000–4000 tokens of input, and prefill is the sequential operation that processes all of that before the first generated token. At 4096 total context, 20 turns of history is mathematically impossible anyway; the overflow silently truncates the oldest turns, creating a confusing agent that appears to forget recent exchanges.

Cutting to max_history_turns = 4 fixed both the stall and the confusion. 4 turns ≈ 400–800 tokens of history — meaningful continuity for a home-control task without dominating the budget. The remaining continuity comes from the memory system, not the raw history buffer. Most home commands don’t need a 20-turn history anyway; they need the right memory injected at the right moment.

Why KV grows faster for transformer-only models

The KV cache growth formula above explains a pattern from the architecture benchmark in genie-ai-runtime: pure transformer models’ KV cache grows proportionally to layers × kv_heads × head_dim, and for the models in that sweep, this means:

• Qwen2.5-1.5B (28 layers, pure attention): KV grows steadily, reaching ~896 MiB at 32k
• Falcon-H1-0.5B (36 layers, parallel attention + Mamba-2 SSM): KV reaches only ~604 MiB at 32k

Why does Falcon-H1 with more layers use less KV memory? Because its Mamba-2 SSM heads don’t contribute to the KV cache at all — SSM state is a fixed-size recurrent state (independent of sequence length), not a growing cache. The KV cache only accumulates for the attention heads, and Falcon-H1-0.5B has fewer attention layers than Qwen2.5-1.5B. The hybrid architecture fundamentally changes the scaling law.

For genie-claw today, this means the 4096-token wall is a property of the transformer models available at the right size and quality. As Mamba-2 and hybrid models mature toward the instruct quality of Qwen3-4B, the context budget opens up — same 8 GB hardware, much longer context. That’s the roadmap.

Accuracy from grounding, not scale

The core agent metric I track is BFCL (Berkeley Function Calling Leaderboard) strict tool-call accuracy. On Jetson Orin Nano 8 GB, Qwen3-4B Q4_K_M, 208 Home Assistant intent test cases:

The 2.5× jump in raw strict is not from a bigger model, more context, or a fine-tune. It’s from injecting the home’s device catalog into the prompt — exact device names, room qualifiers, action vocabulary. Those 300–400 tokens of tool manifest are doing enormous work.

The failure pattern was consistent: the model picked the right tool ~97–98% of the time all along. The failures were in the argument. It emitted "lights" when the catalog says "kitchen lights". The room qualifier was dropped. With the catalog injected, the model copies the exact string from context instead of generating it from memory.

Why this matters for context budgeting. The tool manifest consumes 300–400 tokens — a meaningful fraction of 4096. It would be tempting to trim it to save context budget. These results show that’s exactly the wrong trade: those tokens buy a 2.5× raw accuracy improvement. The context budget isn’t just working memory for the conversation; it’s the source of truth the model reasons from. The tighter the budget, the more you have to think about which grounding information earns its token cost.

The gap between raw 51% and grounded 83% is explained by the runtime resolver: genie-claw already maps "lights" → "kitchen lights" at execution time via resolve_device_alias. Many of those raw-strict failures were calls the agent would actually execute correctly — BFCL penalizes them as wrong because the predicted string doesn’t exactly match the gold label, even if the downstream action is identical. Grounded strict scores both through the same resolver before comparing.

The lesson: for a narrow-domain agent, accuracy comes from deterministic grounding, not model scale. The home’s device catalog is finite and knowable. Ship it in the context window, and the model stops hallucinating room names.

The voice pipeline

genie-claw doesn’t own audio — that’s genie-voice-runtime, a separate service. The current production stack:

ESP32-LyraT mic → I2S2 → deepfilternet denoise (25 dB limit)
  → parakeet.cpp STT (tdt_ctc-110m, ggml CUDA; 1.808 % WER, 61× RTF, +450 MB)
  → genie-claw agent
  → piper TTS (en_US-amy-medium, resident subprocess)
  → speaker

The STT engine is parakeet.cpp — a ggml CUDA build of tdt_ctc-110m that drops in as a resident server on :8178. It beats whisper on every axis that matters for on-device: 1.808 % WER (LibriSpeech test-clean, tdt decoder) at 61× RTF and only +450 MB footprint. The ggml CUDA backend runs 4–14× faster than ONNX Runtime on the same 25 W Orin. The swap is clean — genie-core sees the same POST /inference → {"text": ...} contract; the systemd unit replaces genie-whisper with genie-parakeet.

Hardware frontend: ESP32-LyraT V4.3 (ES8388 codec, 3 onboard mics) wired to Jetson I2S2 at 24 kHz. Total MVP cost is around $550: Orin Nano devkit ($499) + LyraT (~$35) + storage + audio out.

Two hard-won numbers from voice tuning:

• deep_filter_atten_lim_db = 25, not 100 — 100 wiped speech entirely.
• piper_pipe_mode = true — resident TTS subprocess, no per-utterance process respawn. Dropped inter-sentence gap from ~5 s to ~1.5 s.

Total voice round-trip (wake-to-spoken-reply, 3-sentence response): ~22 s after tuning, down from ~52 s. Not instant — 4B model, embedded hardware — but fast enough to feel responsive for home control.

Note that STT’s +450 MB is a real KV cache competitor: every megabyte the audio stack consumes is a megabyte the KV cache can’t use. The system memory budget in the table above is jointly optimized — parakeet’s model choice, the TTS voice size, and the LLM context window are all co-dependent decisions. Switching to whisper-large would add ~800 MB and force either a smaller LLM or a smaller context.

PrivacyProxy: the cloud escape hatch that stays private

The honest limit of a 4B model at 4096 tokens: it handles home control well, but complex tasks — planning, document summarization, multi-step analysis — exceed what a small quantized model does reliably. A 7B+ model breaks the memory budget; a longer context breaks the latency budget.

The escape hatch is PrivacyProxy — an on-device OpenAI-compatible gateway that anonymizes prompts locally before forwarding to a free cloud model (via OpenRouter), then restores real data in the response. The cloud sees __PERSON_1__, __PRIVATE_1__ — never actual names or household details. Detection layers:

1. Deterministic floor (pure Rust, zero external services): private vocabulary, email patterns, US SSN/phone, high-entropy secrets — irreversibly redacted before egress.
2. Optional on-device semantic layer: local LLM for names and orgs the rule set misses (best-effort, not part of the guarantee).

It’s not bulletproof — free-form names the deterministic patterns miss can still go through. The honest framing: it’s a meaningful privacy layer, not a perfect one. For 90% of home commands the local model handles it anyway; PrivacyProxy is the escape hatch for the 10% where a bigger brain helps. Critically, PrivacyProxy doesn’t need to fit in the 8 GB budget — it sends to the cloud. Its only local cost is the anonymization pass (CPU-only, negligible memory).

GeniePod Home: the product

The goal was never a dev-board experiment. The target is a product: GeniePod Home — Jetson Orin Nano Super 8 GB on a custom carrier board (genie-hardware), 4-mic far-field array, NVMe SSD, hardware mic mute, Thread/Matter border router (ESP32-C6 sidecar), RGBW LED status ring. Every layer is custom:

• genie-os — stripped L4T/JetPack image, no desktop services, NVMe mass-flash
• genie-ai-runtime — custom CUDA inference engine replacing llama.cpp
• genie-claw — agent layer: prompt assembly, tool routing, memory, HA integration
• genie-voice-runtime — STT/TTS/audio pipeline
• genie-ai-model — LoRA fine-tunes and a future EAGLE-3 draft head trained on home-traffic patterns

The reason for owning every layer: when memory, latency, and privacy are all hard constraints simultaneously, you can’t afford black boxes anywhere in the stack. If STT takes 200 MB too much, the KV cache loses 200 MB and you lose ~1400 tokens of context budget. If the agent wastes 300 tokens on low-priority memory, the model gets a worse prompt. Every component trades off against every other component, and the only way to tune that is to own the boundaries.

What limited context size taught me

I still think 4096 tokens isn’t enough to build something Jarvis-like — a true conversational partner with long memory and broad reasoning. For that you need longer context, smarter retrieval, or both. The genie-claw roadmap pushes on all of it: Mamba/Falcon-H1 hybrid models for longer context without KV explosion (as shown in the §7 architecture benchmark), better semantic recall for memory that doesn’t fit in the 700-token hydration budget, an EAGLE-3 draft head for speculative decoding on home-traffic patterns to shrink prefill latency.

But the constraint taught something more general: context is the real optimization surface for AI agents, not raw model throughput. The research community optimizes model quality — fine-tunes, bigger reasoning models, longer max context windows. Far fewer people optimize context efficiency: how many useful tokens you can pack into a fixed budget, how much deterministic grounding you can do to let the model focus on reasoning rather than recall, how much history you can compress without losing continuity, which information earns its token cost and which doesn’t.

The numbers are blunt: 300–400 tokens of device catalog injection bought a 2.5× accuracy improvement. 20 turns of history was actively harmful — it burned context budget and caused 50-second prefill stalls. 700 tokens of priority-ordered memory beats 700 tokens of random memory by a large margin. None of this is model architecture. It’s context engineering.

The big cloud players run 128k contexts on H100 clusters, where KV cache is a rounding error in a rack of HBM3. On an 8 GB board in your living room, you figure out what the model actually needs — and build the rest in Rust.