Module 08 — Behavior Preservation: Quality, Fidelity and Acceptance¶
Collection: Hardware-Aware LLM Quantization | Previous: Previous lesson | Next: Next lesson
Three questions need different measurements: does the model solve the task, does it resemble the reference distribution, and does a drafter agree with its target? Calling all three “quality” hides important failures.
1. Define the reference and the observation¶
Use a pinned reference model, tokenizer, chat template, logits processing and context. For teacher-forced fidelity comparisons, both models receive the same prefix tokens. Free-running generations diverge in prefixes, so their next-token KL is a different experiment. Reference fidelity itself does not establish task correctness: the reference can be wrong.
| Question | Useful measures | What the measure does not prove |
|---|---|---|
| Labeled task quality | exact match, task loss, per-slice errors | equivalence to the reference |
| Language-model likelihood | token-weighted NLL/perplexity | every capability or unchanged distribution |
| Reference fidelity | full-vocabulary KL, TV, logit/output error | factual correctness |
| Decisions | top-1 agreement with tie policy | calibrated probabilities |
| Speculative efficiency | accepted draft fraction, emitted tokens/cycle | target task quality |
2. Perplexity and KL, calculated correctly¶
For valid scored tokens, NLL=-sum(log p(token))/N, PPL=exp(NLL). Aggregate token
loss sums and counts, not the arithmetic mean of per-batch perplexities. Shift
causal logits/labels consistently, exclude padding, and state whether prompt
positions are scored. Tokenizers must match before comparing token-level PPL.
For one shared prefix, KL(p||q)=sum_v p_v(log p_v-log q_v). Compute log-softmax in
adequate precision. A top-k API response omits tail mass and cannot directly supply
full-vocabulary KL or a normalized perplexity calculation.
A CPU sketch for causal logits [B,T,V] and a validity mask [B,T] is:
# Both models receive identical input_ids and attention_mask in eval mode.
lp = reference_logits[:, :-1].float().log_softmax(-1)
lq = candidate_logits[:, :-1].float().log_softmax(-1)
valid = attention_mask[:, 1:].bool() & score_mask[:, 1:].bool()
# The caller must also prevent scoring a token across a packed-document boundary.
per_position_kl = (lp.exp() * (lp - lq)).sum(-1)
kl_sum = per_position_kl[valid].double().sum()
count = valid.sum() # accumulate both across batches, then divide; reject count=0
Teacher-forced sequence KL decomposes into expected conditional KL when contexts follow the reference distribution. A convenient corpus average is a useful empirical metric, not automatically that sequence-level expectation.
Why several metrics? Two models can assign the same probability to each observed label and redistribute probability among unobserved tokens: PPL matches, KL need not. Conversely, near a tie, arbitrarily small KL can change top-1. There is no universal “0.01 nats guarantees 99% agreement” threshold.
3. What acceptance actually measures¶
For a fixed prefix, target distribution p, drafter distribution q, and a
proposal x~q, exact speculative sampling accepts with probability
min(1,p(x)/q(x)). Its expected acceptance is:
Terms with q(x)=0 contribute zero; never divide by zero for an unproposed token.
On rejection, sample from normalized max(p-q,0). Accepted mass plus corrected
mass is exactly p. The original algorithm
requires correct distributions and verification/state handling.
This identity relates drafter q to target p, not a quantized target to its
unquantized reference r. Even with a fixed drafter, a change in TV(p,q) does not
identify TV(p,r). If the drafter changes, target-quality inference is weaker still.
Counterexample: r=(0.9,0.1), q=(0.1,0.9). Changing the target from r to q
raises acceptance to 1 while moving it far from r. High acceptance is not fidelity.
4. Tokens emitted per cycle are not a measured TV distance¶
With K proposed tokens and a bonus/correction token, ignoring early EOS and caps:
Only under a homogeneous independent acceptance approximation does this become
1+alpha+...+alpha^K. Runtime counters differ: some report accepted drafts, others
include the bonus. Record the definition. Do not invert a mean acceptance length
and label the resulting effective alpha as measured TV; contexts, correlation,
dynamic K, branches and EOS violate the simplified inversion.
The DSpark case's 2.792→2.546 change is a drafter ablation, not evidence for a 0.082 increase in target-reference TV. Pinned evidence.
5. Uncertainty and evaluation gates¶
The familiar sqrt(alpha(1-alpha)/N) standard error assumes independent Bernoulli
trials. Draft tokens within requests are correlated and later positions are
conditionally observed. Use paired prompts and bootstrap independent requests or
sessions; report prompt counts, seeds, lengths, EOS rules and confidence intervals.
There is no universal 8,000-token sample size that makes every small delta credible.
Set gates from product requirements and a development set, before looking at test results: task accuracy/error limits, critical slices, tail probability changes, context regimes and reproducibility. Keep metrics separate; one scalar cannot certify behavior. A statistically indistinguishable score is not proof of equivalence; predeclare an equivalence/noninferiority margin if that is the intended claim.
Practice and answer guidance¶
Run the CPU contract tests. For p=(0.8,0.2) and
q=(0.2,0.8), acceptance is 0.4. Accepted mass is (0.2,0.2), rejected mass is 0.6,
and the correction distribution is (1,0), yielding (0.8,0.2) exactly.
Returning a fresh p sample after every rejection would yield (0.68,0.32) instead:
that tempting shortcut is biased.
Deliverable: a report with task quality, reference fidelity and draft efficiency in separate columns; identify masks, denominators and independent sampling units.