Lecture 05 — LLM Fundamentals for Agents¶
Course: AI Agent Development 2026 | Previous: Lecture 04 | Next: Lecture 06
Learning Objectives¶
By the end of this lecture you will be able to:
- Explain how transformer inference works at a high level (prefill vs. decode)
- Calculate token counts and context window costs
- Choose the right model for a given agent task
- Understand why latency, throughput, and TTFT matter for agentic loops
1. How Transformers Generate Text¶
An LLM does one thing: given a sequence of tokens, predict the next token. An agent is just a loop that keeps calling this function.
Input tokens → [Transformer] → Logits → Sample → Output token
↓
Append to context
↓
Repeat until stop
Two phases of inference:
| Phase | What happens | Compute bound |
|---|---|---|
| Prefill | Process known prompt positions together, respecting the causal mask | Often compute intensive; depends on shape, batching and kernels |
| Decode | Generate subsequent tokens autoregressively | Often bandwidth limited at small batch; weights, KV/state, communication and compute can each dominate |
Hardware implication: measure the actual workload. Larger batches reuse weights; longer contexts increase attention-state traffic. A roofline estimate is a hypothesis to test, not proof that every prefill saturates compute or every decode is limited only by weights.
2. Tokens and Context Windows¶
Tokens ≠ words. Rough English rules such as four characters per token are budgeting approximations. Code, identifiers, other languages and model tokenizers differ. Count the complete formatted request with the intended model.
import os
import anthropic
client = anthropic.Anthropic()
# Count tokens before sending
response = client.messages.count_tokens(
model=os.environ.get("ANTHROPIC_MODEL", "your-model-id"),
messages=[{"role": "user", "content": "Hello, how are you?"}]
)
print(response.input_tokens) # Actual count depends on model and request formatting.
Context windows change quickly. Think in categories instead of memorizing one snapshot:
| Category | Typical use | Engineering concern |
|---|---|---|
| Small/fast chat model | routing, classification, short summaries | low latency and low cost |
| Balanced agent model | tool use, JSON extraction, code review | reliable structure and good reasoning |
| Long-context model | repo analysis, large documents, multi-file tasks | context cost, retrieval quality, memory pressure |
| Local/open-weight model | edge inference, privacy, offline demos | VRAM, quantization, throughput |
| Embedding model | RAG indexing and retrieval | vector dimension, recall, index cost |
Always verify current context limits in the provider documentation before designing a production agent around a specific window size.
Why context size matters for agents: - Multi-step reasoning accumulates tokens fast - Tool call results land in context - Long documents fed to RAG agent must fit
3. Inference Parameters¶
import os
response = client.messages.create(
model=os.environ.get("ANTHROPIC_MODEL", "your-model-id"),
max_tokens=1024,
# Omit unsupported sampling parameters; use the model-specific contract.
messages=[{"role": "user", "content": "What is 2+2?"}]
)
| Control | Meaning | What to verify |
|---|---|---|
| Sampling parameters | Change token sampling when supported | Supported names, ranges and combinations for the exact model/SDK |
| Output-token limit | Caps generated output | Truncation status, reasoning budget and tool-call completeness |
| Structured output / strict tools | Constrains supported output schemas | Refusal, incomplete output, and local semantic validation |
temperature=0 does not guarantee identical results across kernels, revisions or service executions. It is not a JSON validator. Current Anthropic Python SDK 1.x removes temperature, top_p and top_k from its typed call signature; Claude 4.7+ rejects nondefault values at the API layer. The current model lifecycle guide documents that distinction. Run the provider lab for the pinned SDK contract.
All API examples on this page require a configured supported model and credentials and may incur charges; they were not executed against a hosted model. The fixture lab needs neither.
4. The Anatomy of an API Call¶
import os
import anthropic
client = anthropic.Anthropic() # reads ANTHROPIC_API_KEY from env
response = client.messages.create(
model=os.environ.get("ANTHROPIC_MODEL", "your-model-id"),
max_tokens=2048,
system="You are a helpful assistant.", # system prompt
messages=[
{"role": "user", "content": "Tell me about CUDA."},
{"role": "assistant","content": "CUDA is..."}, # prior turn
{"role": "user", "content": "How does it compare to ROCm?"},
]
)
print("".join(block.text for block in response.content if block.type == "text"))
print(f"Input tokens: {response.usage.input_tokens}")
print(f"Output tokens: {response.usage.output_tokens}")
print(f"Stop reason: {response.stop_reason}") # end_turn | tool_use | max_tokens
stop_reason values for agents:
| Value | Meaning |
|---|---|
end_turn |
Model finished naturally |
tool_use |
Model wants to call a tool — your loop must handle this |
max_tokens |
Incomplete output; never execute a partial tool call |
refusal / pause_turn |
Distinct outcomes requiring an explicit policy; this minimal lab stops |
5. Model Selection for Agent Tasks¶
Not every task needs the most powerful model. Cost and latency add up in multi-step loops.
# Router pattern: use fast/cheap model for simple steps
def route_model(task_type: str) -> str:
fast_model = "provider-fast-model"
balanced_model = "provider-balanced-agent-model"
reasoning_model = "provider-reasoning-model"
routing = {
"classification": fast_model,
"summarization": fast_model,
"tool_use": balanced_model,
"complex_reasoning": reasoning_model,
"coding": balanced_model,
}
return routing.get(task_type, balanced_model)
| Task | Recommended model class | Why |
|---|---|---|
| Simple Q&A, routing | fast model | low latency and cost |
| Tool use, JSON extraction | balanced agent model | reliable structured output |
| Complex reasoning, long context | reasoning or long-context model | stronger planning and larger working set |
| Embeddings | embedding model | specialized vector representation |
6. Streaming for Responsive Agents¶
In agentic UIs, streaming dramatically improves perceived responsiveness.
import os
import anthropic
client = anthropic.Anthropic()
with client.messages.stream(
model=os.environ.get("ANTHROPIC_MODEL", "your-model-id"),
max_tokens=1024,
messages=[{"role": "user", "content": "Explain transformer attention."}]
) as stream:
for text in stream.text_stream:
print(text, end="", flush=True)
# Access final message with usage stats
final = stream.get_final_message()
print(f"\nTokens used: {final.usage.input_tokens} in, {final.usage.output_tokens} out")
7. Cost Estimation¶
# Rough cost calculator.
# Do not hardcode provider prices in production. Load this from a config file
# maintained from the provider pricing page.
PRICING = {
"provider-fast-model": {"input": 0.15, "output": 0.60}, # example only, per 1M tokens
"provider-balanced-agent-model": {"input": 3.00, "output": 15.00},
"provider-reasoning-model": {"input": 15.00, "output": 75.00},
}
def estimate_cost(model: str, input_tokens: int, output_tokens: int) -> float:
p = PRICING[model]
return (input_tokens * p["input"] + output_tokens * p["output"]) / 1_000_000
# A 10-step agent loop with a balanced agent model
steps = 10
per_step_input = 2000 # context grows each step
per_step_output = 500
total = sum(
estimate_cost("provider-balanced-agent-model", per_step_input * i, per_step_output)
for i in range(1, steps + 1)
)
print(f"Estimated loop cost: ${total:.4f}")
Worked answer: these fictional prices and the explicitly assumed 2,000-token growth produce 110,000 input and 5,000 output tokens: $0.330 + $0.075 = $0.405. Actual contexts do not always grow by this amount. Sum actual usage, distinguish cache read/write charges and tool charges, and keep failed attempts in the total. A forecast is not a hard spend cap.
Key Takeaways¶
- Prefill and decode have different reuse patterns; profile before declaring the bottleneck
- Use supported model controls, structured schemas and local validation; sampling is not a correctness guarantee
- Check
stop_reason—tool_usemeans your loop must call the tool and continue - Route tasks to cheaper models where possible; the cost compounds in multi-step loops
- Context grows each step — plan for summarization or windowing in long-running agents
Exercises¶
- Write a script that counts tokens for a 10-page PDF before sending it to the API.
- Build a simple cost logger that wraps
client.messages.createand prints cumulative cost. - Implement a model router that uses a fast model for tasks under 200 input tokens and a balanced agent model otherwise.
Next: Lecture 06