How LLM Tokenization Works (and Why TOON Saves Tokens)

TOON saves tokens because of how LLM tokenization works. Every repeated JSON glyph — {, }, ", :, , — costs tokens on every object in an array. TOON declares field names once in the header, so the per-row structural tax disappears. The result: 39.9% fewer tokens overall and up to 58.8% on flat arrays.

What Is Byte Pair Encoding — the Algorithm Behind Every Major LLM Tokenizer?

Before you can reason about token counts, you need to understand how a tokenizer decides what a "token" is. Every mainstream LLM — GPT, Claude, Gemini, Llama — uses a variant of Byte Pair Encoding (BPE), originally a text compression algorithm described by Philip Gage in 1994 and adapted for neural machine translation by Sennrich, Haddow, and Birch in 2016.

The training procedure: start from raw bytes, count every adjacent pair, merge the most frequent pair into a new token, and repeat until a target vocabulary size is reached. The resulting vocabulary spans single characters up to whole words and common punctuation clusters. At inference time, the same merge rules are applied deterministically — you can predict exactly how any text will tokenize once you know the vocabulary.

How OpenAI's tiktoken o200k_base Tokenizer Works

OpenAI's tiktoken library ships two main vocabularies. The older cl100k_base (used by GPT-3.5 and GPT-4) has roughly 100,000 tokens. The newer o200k_base — used by GPT-4o and the GPT-5 model family — has approximately 200,000 tokens, roughly twice as large. A larger vocabulary means more multi-character sequences are merged into single tokens, which is why GPT-4o and GPT-5 are modestly more token-efficient on natural-language text.

The practical rule of thumb for English text: approximately 4 characters per token. Most common English words tokenize to one or two tokens. Rare words, technical jargon, or strings with unusual character combinations may tokenize to more. Structural punctuation characters — {, }, ", :, , — typically each cost one token because they appear in so many varied contexts that BPE rarely merges them with their neighbors.

That last point is the core of the JSON token-cost problem.

Why JSON Structural Glyphs Are Expensive: A Token-by-Token Breakdown

Consider a single JSON object representing a product in a catalog:

{"id": 42, "name": "Widget A", "price": 9.99, "in_stock": true}

Let us count the tokens. The actual data — 42, Widget A, 9.99, true — accounts for roughly 7 tokens. The structural overhead — the opening and closing braces, four sets of quotes around keys, four colons, three commas, and the key strings themselves (id, name, price, in_stock) — accounts for roughly 20 tokens. You are paying almost 3× more in structural characters and repeated keys than in actual data values.

Now multiply that across an array of N objects with the same schema. Every single object pays for the braces, the quotes, the colons, the commas, and the key names — again, and again, and again. On an array of 100 products, you pay 100 times for the token cost of "id", 100 times for "name", 100 times for "price", 100 times for "in_stock". None of that is data; it is format.

How TOON's Header-Once Design Eliminates the Per-Row Tax

TOON's insight is that for a uniform array of objects — where every row has the same schema — you only need to state the field names once. Here is the same product catalog as TOON:

products[3]{id,name,price,in_stock}:
  42, Widget A, 9.99, true
  43, Widget B, 14.99, false
  44, Widget C, 4.49, true

The field names id, name, price, in_stock appear exactly once, in the header. Each subsequent row is just the values separated by commas — no braces, no quotes around keys, no repeated colons. The LLM reads the header, learns the schema, and can correctly attribute each value to its field on every row. The structural signal is preserved; the structural tax is eliminated.

For comparison, here is the JSON version of the same three rows:

[
  {"id": 42, "name": "Widget A",  "price": 9.99,  "in_stock": true},
  {"id": 43, "name": "Widget B",  "price": 14.99, "in_stock": false},
  {"id": 44, "name": "Widget C",  "price": 4.49,  "in_stock": true}
]

The data is identical. The token cost is not. According to the official toonformat.dev benchmarks — 5,016 LLM calls across 209 questions, six formats, and four models, using the GPT-5 o200k_base tokenizer — TOON uses 58.8% fewer tokens than JSON on flat uniform arrays (67,778 vs 164,452 tokens in the benchmark dataset). Overall across all data shapes, the reduction is 39.9%.

Token Sources in JSON vs TOON: Per-Row Breakdown

The table makes the asymmetry visible: every row in JSON pays for keys, braces, and colons that TOON pays for only once. For a four-field schema, a single JSON row might spend 12–15 tokens on structure for every 7–10 tokens of actual data. In TOON, each data row spends 3 tokens on structure (the commas) for those same 7–10 tokens of data.

Does TOON Save Tokens Equally on All Data Shapes?

No, and being clear about this distinction matters. TOON's savings are tied directly to how repetitive the data schema is across rows. The more uniform the array, the more times the per-row key-and-punctuation cost in JSON repeats, and the larger the relative gain from TOON's header-once design.

The official benchmarks quantify this precisely:

• Flat / uniform tables: 58.8% fewer tokens (67,778 vs 164,452)
• Time-series data (60 days): 59.0% fewer tokens (9,115 vs 22,245)
• GitHub repository data: 42.3% fewer tokens (8,744 vs 15,144)
• E-commerce orders (nested): 33.3% fewer tokens (73,126 vs 109,599)
• Mixed structures: 21.9% fewer tokens (227,830 vs 291,711)

On mixed or highly nested data, the header-once advantage is smaller, and for small payloads the format-instruction overhead — the "prompt tax" — can erase TOON's gains entirely. The arXiv 2603.03306 study formalized this as a scaling hypothesis: TOON pays off only once cumulative per-row savings amortize the upfront overhead — roughly 10–20 objects for a typical four-field schema. For a full treatment, see the JSON vs TOON comparison and the API cost optimization guide.

Token Efficiency and LLM Accuracy: The Two-for-One Gain

Fewer tokens is only half the story. The same toonformat.dev benchmarks that measured token counts also measured retrieval accuracy: TOON achieved 76.4% overall retrieval accuracy versus JSON's 75.0%, and 27.7 accuracy-points per 1,000 tokens versus JSON's 16.4. When the model does not spend attention on repeated keys and structural punctuation, more of its capacity goes to the actual data values and the question — the same mechanism that explains the token savings also explains the accuracy gain. For a comprehensive overview of the format landscape and full syntax reference, see What is TOON? and the TOON specification.

Frequently Asked Questions