Context Window Management: Fitting More In with TOON

TOON helps you fit more into the context window by eliminating repeated JSON structural characters. The official benchmark measured up to 58.8% fewer tokens on flat uniform tables and 39.9% overall — meaning the same window holds roughly twice the rows of uniform data, without any loss in field retrieval accuracy.

Why Context Windows Fill Up So Fast with JSON

Every LLM has a finite context window measured in tokens. You pay for every token in that window on every call — whether it carries information or not. JSON serialization of arrays is structurally expensive: each object in an array repeats every field name, plus opening and closing braces, quotes around keys and string values, and commas between fields. On an array of 500 product records with six fields each, those structural characters account for a large share of the total token count.

A token is roughly 0.75 English words, or about four characters in typical English text. The characters {"product_id":"P-0001","name":"Widget A","price":9.99} tokenize to roughly 20 tokens. Multiply that by 500 rows and you have consumed 10,000 tokens just on one dataset before a single reasoning token is spent. The toonformat.dev benchmark quantified this precisely: the same flat-table dataset took 164,452 tokens as JSON and only 67,778 tokens as TOON — a 58.8% reduction.

For RAG pipelines, agent tool results, and database query responses, the retrieved data is almost always a uniform array of objects. That makes it the highest-value target for context window optimization. See how this plays out in production in Optimizing RAG Pipelines with TOON.

How Many Rows Fit in a Fixed Token Budget?

The following example uses a six-field product record to make the row-count difference concrete. Reserve 2,000 tokens for system prompt and 4,000 tokens for model output; the remaining budget goes to data.

-- Record definition --
Fields: id, sku, name, price, stock, category
Example values: 1001, "WIDGET-A", "Stainless Widget", 29.99, 450, "hardware"

-- JSON serialization (per row) --
{"id":1001,"sku":"WIDGET-A","name":"Stainless Widget","price":29.99,"stock":450,"category":"hardware"}
Tokens per row: ~28 tokens
Plus array punctuation overhead (brackets, commas): ~2 tokens/row
Effective token cost per row in JSON array: ~30 tokens

-- TOON serialization (per row) --
Header (one-time): products[N]{id,sku,name,price,stock,category}:
Header token cost: ~20 tokens  (paid once, not per row)
Each row:          1001, WIDGET-A, Stainless Widget, 29.99, 450, hardware
Tokens per row:    ~12 tokens

-- Rows that fit in a 128k-token window --
Reserve for system prompt + output: 6,000 tokens
Available for data: 122,000 tokens

JSON:  122,000 / 30 tokens per row  = ~4,066 rows
TOON:  (122,000 - 20 header) / 12  = ~10,165 rows

TOON fits roughly 2.5x as many rows in the same window.

-- Rows that fit in a tighter 32k-token window --
Available for data: 26,000 tokens

JSON:  26,000 / 30  =  ~866 rows
TOON:  25,980 / 12  = ~2,165 rows

The multiplier is not exactly 2x in every case because field name length and value length both affect the per-row cost. Longer field names shift the ratio in TOON's favor even further, because TOON declares them once while JSON repeats them on every row. Shorter values with short field names bring the ratio closer to 2x. Use the free converter to measure your actual dataset.

Context Budget vs Rows That Fit: JSON vs TOON

The table below applies the same six-field product record to four common context window sizes. Numbers assume 6,000 tokens reserved for system prompt and output.

Even on a 1M-token window, the absolute row-count difference is enormous: roughly 50,000 additional rows available when using TOON instead of JSON for the same budget. That is 50,000 more product records, log entries, or retrieved passages that the model can reason over in a single call.

Does Packing More Rows Hurt Accuracy?

This is the right question to ask before committing to a denser format. The toonformat.dev benchmark ran 5,016 LLM calls across 209 questions, six formats, and four models (Claude Haiku 4-5, Gemini 3 Flash, GPT-5 Nano, Grok 4.1 Fast). The results are clear for retrieval tasks: TOON achieved 99.6% field retrieval accuracy and 76.4% overall accuracy, modestly above JSON's 75.0%. Higher token density does not degrade comprehension on uniform data.

The accuracy picture changes for computational tasks. TOON's benchmark scores by question type are:

• Field retrieval: 99.6% — essentially perfect
• Structure awareness: 89.0%
• Structural validation: 70.0%
• Aggregation: 61.9% — notably weaker
• Filtering: 56.8% — the weakest category

The implication for context window management is direct: packing a dense TOON block and asking the model to filter or aggregate within it is not reliable. Those operations belong in your application layer, not in the model's attention mechanism on a large dataset.

What to Do When the Data Still Overflows the Window

Switching to TOON is the first lever. If the dataset still does not fit after the token reduction, the correct approach is to reduce the data before it enters the window, not to rely on the model to handle it once it does.

Pre-filter rows

Apply a relevance filter before serialization. For a RAG system, this means your retrieval step already returns only the top-k results matched to the query. For a database query, it means a WHERE clause that eliminates irrelevant rows upstream. Because TOON's filtering accuracy is only 56.8%, asking the model to filter a full dataset after the fact is both less accurate and wasteful of tokens.

Project only the columns you need

TOON's header line makes column projection trivial: change products[N]{id,sku,name,price,stock,category,supplier_id,created_at,updated_at}: to products[N]{id,name,price}: and every row shrinks proportionally. Dropping six of nine fields from each row reduces per-row token cost by roughly two thirds. Do this at the query or serialization layer, not by asking the model to ignore columns.

Summarize or aggregate series before filling the window

Time-series data — logs, metrics, event streams — is a common overflow cause. Rather than passing 60 days of raw daily records, pass the aggregated summary: totals, averages, min/max, and a flag for anomalous days. TOON's aggregation accuracy of 61.9% is a direct argument for doing this computation outside the model. The model's job is reasoning over the result, not re-computing it from raw rows.

Consider TONL for programmatic pre-processing

If your data is large enough that you need a query layer to extract the right slice before filling the context window, TONL is worth evaluating. It provides a SQL-like query API with sub-millisecond indexed lookups and streaming support for files over 50GB in under 100MB of memory. The pattern is: store the full dataset as TONL, query it programmatically to extract the relevant rows, and pass that filtered TOON-compatible slice into the context window. The API cost optimization guide covers how this pattern interacts with caching.

Model Accuracy Varies — Test Before Committing

Overall accuracy numbers mask significant model-to-model variance. On the toonformat.dev benchmark, TOON accuracy ranged from Gemini 3 Flash at 96.7% to GPT-5 Nano at 90.9%, then dropped sharply to Claude Haiku at 59.8% and Grok 4.1 at 58.4%. If you are routing to a smaller or less TOON-familiar model, validate accuracy on your specific dataset and query types before filling the window with a dense TOON block.

The safest approach is to run a parallel evaluation: send the same dataset as JSON and as TOON to your production model, compare accuracy on a representative set of queries, and then make the format decision based on your measured results rather than benchmark averages. The JSON vs TOON comparison walks through exactly this evaluation approach.

Context Window Management and Cost: How They Connect

Every token you remove from the context window saves money on that call and on every subsequent cached re-read. If you are using prompt caching — Anthropic charges cache reads at 10% of the standard input rate; OpenAI caches stable prefixes automatically at roughly 50% of the input rate per the Anthropic caching docs and ngrok's caching overview — a smaller TOON block is cheaper to write into cache and cheaper to read back on every call.

Context window management and cost optimization are the same problem viewed from two angles: fit more useful data per window, pay less per token of that data. For the full cost calculation including batch discounts, see the LLM cost calculator for TOON. For how TOON's token reduction translates into practical architecture decisions for retrieval pipelines, see Optimizing RAG Pipelines with TOON.

Frequently Asked Questions