Lecture 07 — Long context and cache behavior¶
Prerequisites: Lecture 02 and a correct serving baseline. Time: 3 hours. Outcome: separate architectural cache savings from observed runtime allocation.
Model state is not one flat KV tensor¶
The DeepSeek technical report describes cross-layer reuse, hierarchical indexing, FP4 main KV, and bounded replay for sliding-window state. Its reported 890 bytes/token concerns global cache under its architecture accounting. It is not the total memory cost of an arbitrary runtime or a promise that a million-token request fits your node.
Build a ledger for global cache, local/window state, index structures, page metadata, request buffers, vision state, and workspaces. Mark shared versus per-request allocations.
Work the units¶
As a paper-based estimate, 890 bytes/token × 1,048,576 tokens is 890 MiB of global state. Add the other ledger terms and weights before deciding capacity. If your engine uses another representation, the estimate is only a comparison target.
For a generic page allocator, if each page holds P tokens, N requests with lengths L_i consume:
This teaches internal fragmentation. Apply it only to the structures your runtime actually pages; do not assume all compressed V4.1 state follows a conventional decoder-only KV layout.
Controlled sweep¶
- Start with a supported short context and double input length while keeping output length fixed.
- Stop before the predeclared memory or latency limit; do not force an OOM to find a maximum.
- Measure TTFT, decode timing, peak memory per rank, and admitted concurrency.
- Repeat with independent prompts and with a known shared prefix.
- Separate first-use, warm cache, and cache-evicted conditions.
- Verify outputs remain valid as the context grows.
For a reuse experiment, log tokenizer/encoder revision and every prefix-identity input. Different system messages, image content, or settings may alter the computation and invalidate reuse. A higher hit rate is useful only if the returned computation is still correct.
Failure exercise¶
Design two conversations that differ in a meaningful prefix field. Demonstrate that your test would catch a cache erroneously shared between them. Use a local test harness; do not depend on unrelated users' traffic.
Independent transfer: optimize repeated-prefix input-heavy traffic, then re-evaluate a no-reuse workload. Separate improvement due to less work from improvement in kernel efficiency.
Self-check¶
- Why can TTFT rise sharply before memory is exhausted? Chunking, queueing, indexing, and bandwidth can become limiting first.
- Is the maximum configured context a tested capacity? No; it is a configuration ceiling.
- Can per-token memory be inferred from one request? Not reliably: fixed buffers, page rounding, and lazy allocations require a sweep.
Exit artifact: memory/context and latency/context curves, a documented cache-validity test, and a service limit derived from measured behavior. On the analysis route, submit the ledger and equations with unmeasured terms explicit.