Fundamentals — what a token is

What a token actually is

A token is not a character and not a word. It is a subword unit — the atomic chunk a model reads and predicts. Modern models build their vocabulary with Byte-Pair Encoding (BPE): start from individual bytes, then greedily merge the most frequent adjacent pair over and over until you hit a target vocabulary size. Frequent words collapse into one token; rare words fall back to a handful of subword pieces. There is no <unknown> — every possible string is representable, in any language.

Three consequences fall straight out of how BPE works, and they drive the whole course:

• Structured text is expensive. Prose lands near ~4 characters per token; JSON, logs and code are roughly 2× denser because punctuation, indentation and snake_case/camelCase boundaries break merges into many short tokens. Agents traffic almost entirely in structured text — tool calls, file contents, stack traces — so they live in the expensive regime.
• Tokenizers are not interchangeable. Every model family segments differently. You cannot estimate one model's tokens with another's tokenizer — and even within a family it can shift: Anthropic's newer tokenizer (introduced with Opus 4.7) produces roughly 30% more tokens than older models for the same text, and billing follows the new count. Always count with the target model's tokenizer.
• Tokens are the unit of both the bill and the budget. The context window is denominated in tokens; price is per-token; and output costs several times more than input (Opus 4.8: $5/M in, $25/M out — a 5× ratio). Input dominates agent bills anyway, for the reason you'll see next.

Where the tokens go — anatomy of one request

Here is where most cost surprises come from. A single agent turn is not "the user's question." The API is stateless, so every turn re-sends the entire working set. A typical request, before the model has read a single word of your actual task, already carries:

The biggest, most-overlooked sources of waste, with the hardest public numbers we have:

• Tool definitions, repeated every turn. Anthropic has reported tool definitions consuming 134K tokens before optimization, and a typical five-server setup spending ~55K tokens before the conversation even starts. Paid on every single turn.
• Tool-result bloat. Logs, JSON dumps, file contents and search results get appended to history and replayed on every subsequent turn, long after they're stale. Keeping intermediate results out of context cut one Anthropic research workload from 43,588 → 27,297 tokens (−37%).
• History re-send. The whole transcript goes back up the wire each turn — the single largest growing cost, and the subject of Module 2.
• Redundant retrieval. RAG often injects overlapping, low-signal chunks; context is a finite attention budget with diminishing returns, so more tokens can mean worse answers, not just costlier ones.