Chunking Strategies for TOON in RAG Pipelines

For RAG pipelines, chunk uniform retrieved records as TOON tables with the field schema declared once in the header — do filtering and aggregation in the retriever before formatting, not in-context. TOON uses about 59% fewer tokens than JSON on flat uniform arrays, so you fit more evidence rows per context window with overall retrieval accuracy on par with JSON.

Why Chunking Strategy Matters More Than Format Alone

Most RAG guides treat chunking as a purely semantic problem: split at paragraph boundaries, embed the chunks, retrieve the top-k by cosine similarity. Format is treated as an afterthought — usually "just dump the JSON." That works, but it leaves substantial context window capacity on the table.

The format you choose determines how many records fit in the window you hand to the model. The official TOON benchmarks — 5,856 LLM calls across 244 questions, four models, and six formats (v4.1) — show TOON uses about 43% fewer tokens overall versus JSON (42.6%), and 58.7% fewer on flat uniform arrays (68,030 vs 164,643 tokens). In a fixed 128k context window that gap translates directly into more evidence rows per call, which is the core lever in RAG quality.

But token savings alone do not tell the full story. The same benchmark reveals that TOON's accuracy is highly uneven across task types. Routing the right operations to the right layer — retriever versus in-context — is what makes the difference between a RAG pipeline that is merely smaller and one that is actually more accurate.

What TOON Is Good At — and Where It Falls Short

The benchmark breaks accuracy down by question type, and the spread is wide:

• Field retrieval: 97.8% — TOON is near-perfect at reading a specific value from a named field (JSON: 99.2%)
• Structure awareness: 90.3% (JSON: 84.0%) — understanding what fields exist and how the table is organized
• Structural validation: 100% (JSON: 50%; a small category)
• Aggregation: 48.4% (JSON: 48.4%) — counting, summing, or averaging across rows
• Filtering: 38.0% (JSON: 41.1%) — selecting rows that meet a condition

Field retrieval is strong. Filtering and aggregation are not. The practical consequence for RAG is clear: never ask the LLM to filter or aggregate within a TOON chunk. Do that work in the retrieval layer — using a vector DB metadata filter, a SQL WHERE clause, or a pre-aggregation step — then format only the resulting rows as TOON. The model then receives a clean, pre-filtered table and only needs to do what it does well: read field values and reason about their meaning.

Overall retrieval accuracy is 72.2% for TOON versus 71.4% for JSON — effectively tied — with TOON delivering 29.2 accuracy-points per 1,000 tokens versus JSON's 16.6. That efficiency ratio is why TOON belongs in the formatting layer of a RAG pipeline, not just as a smaller version of JSON.

JSON Array vs TOON Table: What a Retrieved Chunk Looks Like

Consider a retriever that returns six product records. Here is what those records look like as a standard JSON array chunk versus a TOON table chunk:

// JSON array chunk — field names repeated on every object
// ~320 tokens for 6 records
[
  {"id": "p-101", "name": "Wireless Keyboard", "price": 49.99, "stock": 214, "category": "peripherals"},
  {"id": "p-102", "name": "USB-C Hub 7-in-1", "price": 34.99, "stock": 89, "category": "peripherals"},
  {"id": "p-103", "name": "Monitor Stand", "price": 29.99, "stock": 57, "category": "accessories"},
  {"id": "p-104", "name": "Webcam 1080p", "price": 69.99, "stock": 132, "category": "peripherals"},
  {"id": "p-105", "name": "Desk Lamp LED", "price": 24.99, "stock": 301, "category": "accessories"},
  {"id": "p-106", "name": "Laptop Stand", "price": 39.99, "stock": 44, "category": "accessories"}
]

// TOON table chunk — field schema declared once in the header
// ~130 tokens for the same 6 records (~59% fewer in this example)
products[6]{id,name,price,stock,category}:
  p-101, Wireless Keyboard, 49.99, 214, peripherals
  p-102, USB-C Hub 7-in-1, 34.99, 89, peripherals
  p-103, Monitor Stand, 29.99, 57, accessories
  p-104, Webcam 1080p, 69.99, 132, peripherals
  p-105, Desk Lamp LED, 24.99, 301, accessories
  p-106, Laptop Stand, 39.99, 44, accessories

The header line — products[6]{id,name,price,stock,category}: — does three things at once: declares the array name, signals the expected row count so the model can detect truncation, and names every field. The model reads one header and then processes pure values, which is why field retrieval accuracy stays at 97.8% despite the much smaller payload.

Critically, filtering has already happened before this chunk was built. If the user query only concerns peripherals, the retriever filtered on category = peripherals first, and only the matching rows were formatted as TOON. The LLM never sees the accessories rows and never needs to filter.

JSON Chunks vs TOON Chunks: A Direct Comparison

The following table uses figures derived from the official TOON benchmarks to show how the two chunk formats compare across the metrics that matter most in RAG:

Token savings on e-commerce data with nesting are smaller — 32.9% — but still meaningful. The mixed-structure track as a whole yields 32.7%. The chunking benefit of TOON is strongest when your retrieved records share a uniform schema, which is the common case for database-backed RAG (product catalogs, user records, log entries, time-series readings).

How to Structure the RAG Retrieval Layer for TOON

The recommended pipeline has four stages, keeping each operation in the layer that handles it best:

Stage 1 — Semantic retrieval. Use your vector database to retrieve the top-k candidate records by embedding similarity. Do not worry about format yet; retrieve raw records.

Stage 2 — Pre-filter and pre-aggregate. Apply any WHERE-style filters and aggregation functions in the retrieval layer using native database capabilities (SQL, metadata filters, faceted search). If the user asks for "products under $50 in stock," filter on price and stock before touching the LLM. This is where the 38.0% and 48.4% accuracy risks live — eliminate them here.

Stage 3 — Format as TOON. Take the filtered result set and serialize it as a TOON table. Group records by schema: one TOON block per record type. If you retrieved products and also reviews, emit two separate TOON blocks rather than mixing schemas in one block.

Stage 4 — In-context reasoning. The LLM receives a clean, pre-filtered TOON table and answers field-retrieval and reasoning questions. It does not need to filter, count, or sum — it needs to read and reason, which is where TOON's 97.8% field-retrieval accuracy shines.

For a broader treatment of TOON in retrieval-augmented generation architectures, see our guide on optimizing RAG pipelines with TOON.

Keeping the Field Schema in the Header

The most important structural rule for TOON chunks is: one schema, one header, one block. The header line records[n]{field1,field2,field3}: gives the model an explicit count and field list it can use to validate completeness. If a chunk is truncated mid-table, the model can detect the mismatch between the declared count and the rows it received.

This explicit schema also means you do not need to include a separate "schema description" in your system prompt for each record type — the schema travels with the data. That reduces per-call prompt overhead, which compounds across thousands of RAG calls.

One practical tip: keep the field list in the header in the same order as the values in each row, and use short but recognizable field names. TOON's structure-awareness score of 90.3% means the model understands the table layout well, but ambiguous abbreviated field names can reduce that further. Spell out unit_price rather than uprc.

For questions about embedding data in formats other than TOON, including binary and embedding vector representations, see embeddings and binary data in LLM formats.

Model Compatibility and When to Fall Back

TOON accuracy is not uniform across models. The official v4.1 benchmark records Grok 4.5 at 97.1%, Gemini 3.6 Flash at 69.3%, Claude Haiku 4.5 at 65.6%, and GPT-5.4 Nano at 57.0% with TOON. JSON lands within about two points of TOON on every model, so the format choice mostly affects cost, not accuracy — but on smaller models TOON's small edge can disappear (GPT-5.4 Nano scored 57.4% with JSON).

Before deploying TOON chunking in production, run a sample of your actual queries against both JSON and TOON chunks using the model you are targeting. If you are on GPT-5.4 Nano or a similarly rated model, you may find that JSON chunks deliver comparable retrieval accuracy without requiring format instructions. The TOON overview covers model compatibility in more detail.

For production pipelines where you need schema validation, streaming for large result sets, or a query API, consider TOON best practices or evaluate TONL, which adds those capabilities while still saving 32–45% of tokens versus JSON.

Frequently Asked Questions