Tokenization and the Cost of a Request

Before a model can predict anything, whatever you type has to be converted into a form it can actually read. This tutorial starts here, one level below the conversation itself: how text becomes the individual pieces, tokens, a model processes one at a time, and, since every hosted model bills per token rather than per character or word, what a request built out of them actually costs. Chapter 2 picks up right where this leaves off: an actual conversation with a model, generated one of these tokens at a time.

1.1 Byte-Pair Encoding

Every modern LLM tokenizer, including the ones behind GPT and Claude, is a variant of byte-pair encoding (BPE). The algorithm starts from individual characters and repeatedly merges whichever adjacent pair of symbols occurs most often across a training corpus into a new symbol, generation after generation. Common words end up collapsing into a single token; words the tokenizer never saw during training (a typo, a made-up word, a rare technical term) fall back to smaller, still-meaningful pieces instead of one unhelpful "unknown word" marker.

No provider publishes the exact vocabulary their production tokenizer learned this way, so there's no way to train a small local copy and expect its token boundaries to line up with a real model's. The demo above sidesteps that entirely: instead of approximating, it asks Claude directly for the exact count of whichever text you enter, the same count that model would actually bill for.

1.2 Turning Tokens into Dollars

Once text is tokenized, cost is just arithmetic: providers publish a price per million tokens, separately for input (what you send) and output (what the model generates), and output is typically priced several times higher than input. At real, published rates, a single short request costs a fraction of a cent, but that number stops being trivial the moment a feature is handling thousands of requests a day, which is exactly the "cost optimization" concern Chapters 6 and 7 pick up next: how to spend fewer tokens without spending less accuracy.