TOON Benchmarks 2026: Token Savings and Accuracy by Model

The current official TOON benchmark (v4.1, 5,856 LLM calls across four models) shows TOON using 42.6% fewer tokens than JSON while scoring 72.2% retrieval accuracy vs JSON's 71.4%. Accuracy is effectively tied. The win is efficiency: 29.2 accuracy points per 1,000 tokens, compared to JSON's 16.6.

Benchmark Methodology: What Was Tested

The benchmark is reproducible and fully documented at toonformat.dev/guide/benchmarks, with raw results in the retrieval-accuracy and token-efficiency reports. The test matrix covered:

• 244 questions per format per model, over 13 datasets
• 6 formats tested head-to-head: TOON, JSON (pretty), JSON compact, YAML, XML and CSV (CSV only answers the 109 flat-data questions)
• 4 models: claude-haiku-4-5-20251001, gemini-3.6-flash, gpt-5.4-nano, grok-4.5
• 5,856 total LLM calls (244 × 6 × 4)
• Tokenized with the GPT-5 o200k_base encoding via gpt-tokenizer; reasoning disabled, temperature left at model defaults
• Accuracy reported with Wilson 95% confidence intervals

Questions span five types: field retrieval, aggregation, filtering, structure awareness and structural validation. The datasets include uniform employee records, time-series analytics, GitHub repositories, nested e-commerce orders, semi-uniform event logs, deeply nested configuration, five deliberately corrupted validation sets, and two datasets new in the v4 run: feature flags keyed by name and contacts with nested address and plan groups.

v4 note: those two new datasets exercise forms added in TOON spec 4.0. Keyed tables (flags[3:]{enabled,rollout}:) turn an object of uniform objects into a table, and nested field groups (orders[2]{id,customer{name,country},total}:) keep arrays with uniform nested objects tabular. Spec 4.0 also added full-line # comments. Earlier runs had no equivalent data, which is part of why the headline numbers moved.

About the old numbers: an earlier run (early 2026) reported 39.9% fewer tokens and 76.4% vs 75.0% accuracy over 5,016 calls, with different models (Gemini 3 Flash, GPT-5 Nano, Claude Haiku 4.5, Grok 4.1). Those figures are superseded. Every number in this article comes from the current v4.1 run.

The Headline Number: TOON vs JSON Side by Side

Before looking at the tables, it helps to see why token counts diverge. Here is the same three-user array in JSON and then in TOON, as produced by encode() from @toon-format/toon 4.1.1:

{
  "users": [
    { "id": 1, "name": "Alice", "role": "admin", "active": true },
    { "id": 2, "name": "Bob", "role": "viewer", "active": true },
    { "id": 3, "name": "Carol", "role": "editor", "active": false }
  ]
}

users[3]{id,name,role,active}:
  1,Alice,admin,true
  2,Bob,viewer,true
  3,Carol,editor,false

Pretty-printed with two-space indentation, the JSON is 305 characters; minified it is 171. The TOON version is 94 characters. Every key is written once instead of once per row, and the rows carry no quotes or braces. The header line users[3]{id,name,role,active}: also gives the model an explicit row count and schema to check against, which is why TOON does so well on the structural-validation questions below.

For a deeper look at the structural differences, see What is TOON? and the full JSON vs TOON comparison.

How Much Does TOON Save? Token Reduction by Data Shape

Token savings are not uniform. They depend on how repetitive the data is. The benchmark splits datasets into a flat-only track (fully tabular, CSV-compatible) and a mixed-structure track (nested or semi-uniform). All figures come from the official token-efficiency results using the o200k_base tokenizer.

The pattern is clear: TOON's savings track key repetition. A flat table of employee records repeats the same field names on every row in JSON; TOON declares them once. Against minified JSON the picture is more honest. TOON still saves 35.2% on flat data, but on the mixed track it is 1.6% larger overall, and 19.9% larger on semi-uniform event logs where only half the arrays qualify for tabular form.

On the flat track, CSV is the smallest format of all: TOON is 5.9% larger than CSV (68,030 vs 64,247 tokens), the cost of its array lengths and field headers. Want to see how TOON compares against YAML, CSV and TOML in more depth? See the full format comparison.

Accuracy by Model: Which LLM Reads TOON Best?

Token savings only matter if the model can still answer correctly. All four models answered the same 244 questions in every format. Source: official retrieval-accuracy results.

The spread between models (57% to 97%) is far larger than the spread between formats on any single model (at most 2.1 points between TOON and JSON). TOON is ahead on three of the four models, but on every model the TOON and JSON confidence intervals overlap. The benchmark authors are explicit that when intervals overlap, "the difference between them is not statistically meaningful." The practical reading: switching to TOON does not cost accuracy on any of these models, and choosing a stronger model matters much more than choosing a format. Still, benchmark your own workload before committing at scale.

Accuracy by Question Type: Where TOON Excels and Where It Struggles

Aggregate accuracy hides real variation across task types. Here is TOON next to JSON, averaged over all four models:

Structural validation is TOON's standout. The datasets include truncated arrays, extra rows, a missing field and missing required values. TOON's declared length and field header (users[3]{...}:) let the model spot the mismatch, while JSON offers nothing to compare against. Treat the 100% vs 50% gap with care, though: the category has only five questions (one per corrupted dataset). Structure awareness benefits from the same mechanism, since the header states the row count directly.

Field retrieval is near-perfect for every format; JSON is marginally ahead. Aggregation and filtering are hard everywhere because they require the model to scan and compute across many rows, and TOON is slightly behind JSON on filtering. For workloads heavy in aggregation or filtering, do the math in code and send the model the result. See how to reduce LLM API costs in production.

The Efficiency Metric: Accuracy per 1,000 Tokens

Raw accuracy alone is misleading if two formats have different token costs. The benchmark computes a composite efficiency score: accuracy percentage per 1,000 tokens consumed (average tokens per question context).

TOON matches JSON's accuracy while using 42.6% fewer tokens. At equal budget, that means you can send roughly 74% more data (4,308 ÷ 2,474) without losing accuracy. On the 109 flat-data questions, where CSV can compete, the result is similar: TOON 63.1% at 1,994 tokens, CSV 62.2% at 1,851, and JSON 60.3% at 3,950. For RAG pipelines that retrieve dozens of records per query, efficiency is the number that matters most. See how this plays out in the JSON vs TOON deep-dive.

When TOON Does Not Beat JSON: The "Prompt Tax" and Non-Linear Payoff

No benchmark summary is complete without the caveats. An independent paper, "Token-Oriented Object Notation vs JSON: A Benchmark of Plain and Constrained Decoding Generation" (arXiv 2603.03306, Feb 2026), and the official results themselves point to conditions where JSON wins or ties:

1. Generation tasks. When the model must output TOON rather than read it, plain JSON achieves better one-shot and final accuracy. The reason is the "prompt tax": to reliably produce valid TOON, you must include format instructions in the system prompt. Those instructions consume tokens, partially offsetting TOON's savings, and add a failure mode if the model ignores them.
2. Small payloads. TOON's savings are non-linear. The header (fieldName[n]{col1,col2,...}:) has a fixed cost, and on a three-row array the savings are small in absolute terms. If you also need format instructions, they can cost more than you save. As a rule of thumb (ours, not the benchmark's), the payoff becomes clear once you have a few dozen rows of uniform data.
3. Semi-uniform and deeply nested data. Against minified JSON, TOON is 19.9% larger on semi-uniform event logs and 6.7% larger on deeply nested configuration. If your payload looks like that, compact JSON is often the better choice.
4. Multi-turn agent loops. A separate study (arXiv 2605.29676) on agentic tool-use benchmarks with five open-weight models found TOON saved up to 18% of tokens at roughly a 9-point accuracy cost, and concluded it is "not safe as a default in multi-turn agentic systems."

Model choice also matters far more than format: the same TOON input scores 57.0% on gpt-5.4-nano and 97.1% on grok-4.5. Always test the specific model, task type and data shape for your production workload before switching formats at scale. Our guide on when not to use TOON covers these cases in more detail.

What This Means in Production

The benchmark data supports a practical decision framework:

• Use TOON when: injecting large uniform arrays (logs with a fixed schema, time-series, product catalogs, user records), maps of uniform objects (keyed tables) or records with uniform nested objects (nested field groups) into a prompt. Expect 40–67% fewer tokens than pretty JSON with accuracy on par, across all four tested models.
• Use JSON when: the model must generate structured output, the payload is tiny, or the data is semi-uniform or deeply heterogeneous. In those cases minified JSON is often as small or smaller.
• Use CSV when: the data is purely flat and you do not need the length and header guardrails. It is about 6% smaller than TOON on flat tables.
• Test first when: your workload is mostly aggregation or filtering questions, where every format scores below 50%.

To convert your existing JSON to TOON instantly without writing any code, use the free JSON to TOON converter. For a comprehensive look at which format fits which scenario, read the TOON format comparison guide.

Frequently Asked Questions