What a Token Actually Is, and Why a UUID Costs 24 of Them

There’s a line of code in a lot of AI products that looks like this:

estimated_tokens = len(text) / 4

I measured it against real encodings. It came out 30% too high on formal English, 33% too low on indented JSON, 61% too low on UUIDs and 70% too low on Japanese.

Those errors have a cause, and it explains most of what’s surprising about token costs. So I built the tokenizer first.

A tokenizer is a compression table

Most LLMs with a public tokenizer use a variant of byte pair encoding. Sennrich, Haddow and Birch brought it to language modelling in 2016 as a way to handle rare words. The algorithm fits on one screen:

def train(text, vocab_size):
    ids = list(text.encode("utf-8"))    # start with one id per byte: 0-255
    merges = {}
    for new_id in range(256, vocab_size):
        pair, count = Counter(zip(ids, ids[1:])).most_common(1)[0]
        if count < 2:
            break
        merges[pair] = new_id           # this pair of ids is now a single id
        ids = merge(ids, pair, new_id)
    return merges

Count adjacent pairs, merge the most common one, repeat. Encoding replays those merges in the order they were learned. That’s the whole idea. Production tokenizers add a pre-split on word boundaries, a set of special tokens and a lot of speed work on top.

Notice what that means. A token is a string that was common in the training data. Your bill depends on how much your text resembles whatever the vocabulary was trained on.

I trained a toy 1,024-entry tokenizer on nothing but the English text of the Universal Declaration of Human Rights, then pointed it at other content:

Content	Bytes per token
English UDHR (its own training text)	4.63
Spanish UDHR	1.79
Python source	1.51
UUIDs	1.02
Tamil UDHR	1.01

It compresses what it has seen and falls back to single bytes for everything else. A UUID is random hex with no common pairs to merge, so it barely compresses at all. Real vocabularies are vastly larger (100,277 entries in cl100k_base, 200,019 in o200k_base) and trained on far more text, but the shape is the same.

What content actually costs

From here on these are real encodings, measured with tiktoken: cl100k_base is the encoding GPT-4 used, and o200k_base is the one GPT-4o moved to.

Start with something every agent builder should care about, because tool results are mostly structured data. I rendered the same 200 records six ways:

Format	Tokens	vs CSV
XML	9,047	3.47×
JSON, indented	9,043	3.47×
YAML	6,575	2.52×
JSON, minified	5,243	2.01×
Markdown table	3,399	1.30×
CSV	2,608	1.00×

The expensive part is repetition you can’t see. Indented JSON repeats every key and all its punctuation on every row, and every newline-plus-spaces run has to be paid for. json.dumps(data, indent=2) in a tool handler costs 72% more than the same call with compact separators, for no benefit to the model that I can think of.

YAML is the surprise. People reach for it to save tokens, and here it costs more than minified JSON.

Now identifiers, which is the title of this post:

One item	Tokens each
base64 of 48 bytes	44.8
SHA-256 hex digest	37.8
UUIDv4	23.8
ISO-8601 timestamp with milliseconds	16.0
IPv4 address	8.0
A short alias like t17	3.0

Twenty UUIDs in a tool result is about 476 tokens of pure entropy. Worse, the model can’t do anything useful with them except copy them back, and copying 36 random characters is exactly where you don’t want a sampling error.

Then languages, where the generational change is striking. These are characters per token over the first 10 KB of the UDHR in each language. The corpus truncates by bytes, so compare the rates and not the totals.

Language	cl100k_base	o200k_base	Tokens saved by newer encoding
English	5.19	5.19	none
Spanish	3.95	4.74	1.20×
German	3.57	4.59	1.28×
Russian	2.25	3.97	1.76×
Arabic	1.44	3.12	2.17×
Korean	1.04	1.75	1.68×
Chinese	1.00	1.42	1.42×
Hindi	1.02	3.29	3.23×
Japanese	0.90	1.21	1.35×
Tamil	0.73	2.78	3.82×

Under the older encoding, Tamil used more tokens than characters. English didn’t change at all between generations, so if you only test in English, you’d never have noticed that a vocabulary upgrade cut your Tamil users’ costs by almost four times.

Tokens are not a stable unit

The Claude documentation carries a note worth reading twice. Claude 4.7 and later use a newer tokenizer, and the same input produces approximately 30 percent more tokens than on earlier models. A model migration can change your token count, your context-window fit and your bill without a single prompt changing.

I can’t give you Claude numbers here because this run used only public encodings. For Claude, the same page describes a count_tokens endpoint that’s free to use and counts under the tokenizer of the model you name. It returns an estimate, which is still a far better one than len / 4.

What I’d change in a codebase

Take len / 4 out of anything that enforces a limit. It’s fine for a napkin. It’s wrong in opposite directions depending on the content, so a limit built on it can truncate English early and blow past the budget on JSON.

Minify JSON in tool results today. It’s a one-argument change, and 42% of the tokens go away.

Try CSV for flat tabular tool output, and run your evals before you keep it. Compact formats save tokens by removing structure the model might be using. I measured cost here, not comprehension.

Alias long identifiers in the harness. Map t17 to the real UUID in code and restore it when a tool call comes back. You save about 21 tokens per ID, and an alias the harness never issued fails cleanly at lookup instead of becoming a plausible-looking UUID.

Budget per language. If you have one global token limit per user, your Japanese users get less than a quarter of the content your English users do.

Recount on every model migration, with that model’s own counter. Treat it like the certificate-expiry problem it is, and keep an eye on the effect on your token budget and your prompt cache boundaries.

Limits of this measurement

The English sample is formal legal prose, which flatters the tokenizer; casual text and code will land elsewhere (Python source came in at 5.03 characters per token here). The records are short and flat. I measured input encoding only: no images, no thinking tokens, and nothing about how well a model reads each format.

Run it

Everything is in ajinb/tokens-from-scratch: the BPE implementation with round-trip tests over arbitrary Unicode, and the script that produced every table above. Point measure.py at your own tool outputs. The result is usually a better argument for a one-line fix than anything I can write here.