Attention correctness: negative scores and growing KV prefixes¶
Purpose: connect the mathematical attention operation to the bytes actually written by a decode loop. Prerequisites: arrays, softmax, and the runtime CPU lab. Layers and roles: inference kernels, runtime memory, and numerical validation; preparation for inference and accelerator engineering. Allow 20 minutes for the worked examples, 30 for the CPU checks, and 30 for a fault explanation. The optional GPU setup is separate.
This lesson corrects two defects in the experimental runtime: floating maxima were reduced with a signed-integer comparison, and the attention reader assumed a different KV layout from the token writer. It does not establish that the runtime can generate correct text from a pretrained checkpoint. Other kernels, model conventions, and end-to-end execution still need target validation.
1. Negative logits are ordinary inputs¶
For one query and two keys, suppose the scaled scores are [-1000, -1001] and
the values are [2,10]. Subtract the maximum before exponentiation:
Computing exp(-1000) directly underflows in ordinary floating arithmetic, so
the shifted expression is essential. A GPU block must also find the correct
maximum. Reinterpreting a negative IEEE-754 float's bits as a signed integer
does not preserve numerical ordering. In particular, the integer encoding
of -FLT_MAX exceeds the encoding of -1000. The old integer atomicMax
could retain the sentinel and produce invalid exponentials for an entirely
negative tile.
The corrected kernel computes floating maxima inside each warp, stores four warp results, and combines them with floating comparisons. It also combines the four partial sums in a fixed order. This is a correctness choice, not a claim that this reduction is the fastest implementation on a Jetson.
2. Current length is not a storage stride¶
The pool reserves separate K and V spans per layer. Each span is
[capacity, kv_heads, head_dim], with dimension contiguous:
With capacity 7, two KV heads, three dimensions, and FP16 storage, each token
occupies 2 × 3 × 2 = 12 bytes in K and another 12 in V. V begins 7 × 12 = 84
bytes after K for that layer. The current length selects how many token rows to
read; it never changes these addresses.
The old reader instead used kv_head * current_length * head_dim + ..., treating
the same bytes as head-major storage. The error can stay hidden when there is
only one KV head or one token. A robust test therefore needs multiple heads,
multiple tokens, and a capacity larger than the active prefix.
For a four-query-head/two-KV-head example, query heads 0–1 share KV head 0 and
heads 2–3 share KV head 1. Set all queries to zero, so attention weights are
uniform. Let V[t,h,d] = 100h + 10t + d for three tokens. The correct outputs
are [10,11,12] for query heads 0–1 and [110,111,112] for heads 2–3.
This known answer catches a layout error without depending on a second softmax
implementation.
3. Preserve the online softmax invariant¶
For previously seen keys, retain maximum m, denominator l, and unnormalized
weighted values a. A new tile raises the maximum to m_new. Before adding
that tile, rescale both old accumulators:
Then add exp(score - m_new) to the denominator and its weighted value to the
numerator. The final result is a/l. Rescaling only the denominator corrupts
the output when a later tile contains the largest score. The CPU exercise
compares this tiled mathematical model with a separately materialized-score
double-precision oracle, using five tile sizes for every fixture.
4. Execute the CPU checks¶
Run from Projects/jetson-llm-runtime:
This executed test compiles the actual pool pointer logic with the bounded host CUDA-allocation shim and runs the C++ reference executable. It covers:
- 13 fixtures × 5 tile sizes, including negative logits, GQA, partial tiles, head dimension 129, and prefix lengths 1, 17, 63, 64, 65, and 129.
- Independent known answers for two-key attention, uniform GQA attention, and the single-token case.
- The original signed-integer comparison counterexample.
- Actual
KVCachePooloffsets versus the shared token-major address helper, across two layers and separate K/V spans. - Invalid GQA geometry, empty prefixes, excessive head dimensions, and invalid scale values.
A second native test checks the actual scratch allocator and decode planner: known peak sizes, host-buffer reservation before KV admission, checked arithmetic, 200 layer allocation/rewind cycles with preserved token state, and rejection of invalid checkpoints or exhausted storage. See the worked scratch budget. This host trace does not execute CUDA stream scheduling.
The test passes on the recorded x86-64 CPU host. A separate ASan/UBSan build also
passes. These results check CPU math, address arithmetic, and host pool control
logic. They do not execute CUDA shuffles, GPU memory transactions, or a model.
The normal CPU CMake build also exposes the attention_cpu_oracle CTest target.
5. Compile and, on a compatible GPU, run parity¶
The optional harness links the actual attention kernel directly. From the same project directory, using a working CUDA toolkit:
nvcc -std=c++17 -arch=sm_87 -O2 -I include \
tests/test_attention.cu src/kernels/attention.cu -o /tmp/jllm-attention-parity
/tmp/jllm-attention-parity
Recorded status: compilation succeeded with NVCC 13.4.92, targeting
SM 8.7. Execution returned 77 with “no usable CUDA device/driver”; GPU
parity was not executed. Compilation on this x86-64 host is not a Jetson ARM64
binary or evidence of JetPack compatibility. Build with the installed supported
target toolchain before target measurements. The separate decode.cu translation
unit also compiled for SM 8.7.
On a usable GPU, the harness poisons output with NaNs before every invocation,
checks CUDA launch/completion/copy errors, rejects nonfinite outputs, and compares
against the double-precision oracle. The fixed acceptance rule is
abs(error) <= 0.003 + 0.001 * abs(reference), allowing FP16 output rounding in
these bounded fixtures. It reports each maximum absolute error. Unused FP16 cache
capacity contains NaNs to expose reads beyond the active data.
The isolated kernel also has one INT8 fixture with explicit fixed per-head K and V scales. This does not enable INT8 caching in the runtime: append-time quantization, scale storage, calibration, and model-quality parity remain unsupported. The runtime rejects INT8 and CUDA-graph generation modes. A missing scale pointer is now an invalid kernel call rather than an implicit scale of 1.
6. Completion and transfer¶
Save your CPU output and annotate one cache row by hand. Explain why length 1 can conceal the old layout error, why a new tile maximum changes the old numerator, and why a successful compiler exit is not a numerical result. For an independent variation, change the GQA group size and active prefix while retaining the same backing storage. Predict a uniform-attention answer before running it. Keep the optional GPU result labeled “not run” until it actually runs.
The validation record contains source/binary hashes, commands, CPU results, compiler version, and the explicit GPU skip. No throughput, energy, memory-bandwidth, pretrained-text quality, or Jetson execution result is claimed by this lesson.