TOON as the Wire Format for Multi-Agent Systems

For multi-agent systems, use TOON for shared context and tabular intermediate results, and JSON for structured agent output. TOON cuts tokens by 39.9% on average — and because shared state is re-tokenized at every hop, that saving compounds across the full pipeline. JSON stays the right choice when an agent must generate structured commands or tool arguments.

Why Format Overhead Compounds Across Agent Hops

In a single LLM call, choosing TOON over JSON is a straightforward token-budget decision. In a multi-agent system, the economics change: the same shared context — retrieved rows, tool results, intermediate state — is re-sent with every agent invocation that needs it. If three agents each consume a 1,000-token payload, you pay 3,000 tokens for that payload. If TOON reduces that payload to 600 tokens, you pay 1,800 tokens across the same three hops — a 1,200-token saving that would have been only 400 tokens on a single call.

The official toonformat.dev benchmarks measured 5,016 LLM calls across 209 questions, six formats, and four models. The result: TOON achieved 76.4% retrieval accuracy versus JSON's 75.0%, while using 39.9% fewer tokens. On flat uniform tables the reduction reached 58.8% (67,778 versus 164,452 tokens). Those per-message figures multiply by the number of hops where the same data flows.

This compounding is the strongest argument for TOON as a wire format in multi-agent architectures. It is not a marginal optimization — on a four-agent pipeline processing tabular data, the cost difference between JSON and TOON can approach 50% of total input-token spend.

Side-by-Side: Shared Agent State as JSON vs TOON

Consider a retrieval agent that fetches recent orders and passes them to a summarizer agent and a pricing agent. Here is the same intermediate result in both formats:

// JSON — shared state passed to downstream agents
// Keys repeated on every object: ~310 tokens for 8 rows
{
  "orders": [
    {"id": "ORD-001", "sku": "WIDGET-A", "qty": 12, "unit_price": 9.99, "status": "shipped"},
    {"id": "ORD-002", "sku": "GADGET-B", "qty": 4,  "unit_price": 24.50, "status": "pending"},
    {"id": "ORD-003", "sku": "WIDGET-A", "qty": 30, "unit_price": 9.99, "status": "shipped"},
    {"id": "ORD-004", "sku": "PART-C",   "qty": 1,  "unit_price": 149.00, "status": "processing"},
    {"id": "ORD-005", "sku": "GADGET-B", "qty": 8,  "unit_price": 24.50, "status": "shipped"},
    {"id": "ORD-006", "sku": "WIDGET-A", "qty": 5,  "unit_price": 9.99, "status": "pending"},
    {"id": "ORD-007", "sku": "PART-C",   "qty": 2,  "unit_price": 149.00, "status": "shipped"},
    {"id": "ORD-008", "sku": "GADGET-B", "qty": 6,  "unit_price": 24.50, "status": "processing"}
  ]
}

// TOON — same data, ~33% fewer tokens; fields declared once
// Re-sent to every downstream agent at this reduced cost
orders[8]{id,sku,qty,unit_price,status}:
  ORD-001, WIDGET-A, 12, 9.99, shipped
  ORD-002, GADGET-B, 4, 24.50, pending
  ORD-003, WIDGET-A, 30, 9.99, shipped
  ORD-004, PART-C, 1, 149.00, processing
  ORD-005, GADGET-B, 8, 24.50, shipped
  ORD-006, WIDGET-A, 5, 9.99, pending
  ORD-007, PART-C, 2, 149.00, shipped
  ORD-008, GADGET-B, 6, 24.50, processing

When this payload flows to a summarizer agent and a pricing agent, the TOON saving applies twice. On a 10-agent fan-out, it applies ten times. The JSON keys id, sku, qty, unit_price, and status — each repeated eight times, each quoted — are pure overhead that TOON eliminates by declaring them once in the header line.

JSON vs TOON Wire Format Across a Multi-Hop Pipeline

The table below models a four-agent pipeline (retrieval → enrichment → summarizer → auditor) passing 50 order rows at each step. Token counts are estimated from the benchmark data; the compounding column shows cumulative input-token spend by the final hop.

Notice the final row: the auditor's structured output uses JSON. That is not an oversight — it reflects a real constraint from independent research.

When to Keep JSON: Agent Output and Structured Commands

A February 2026 arXiv paper — arXiv 2603.03306, "Token-Oriented Object Notation vs JSON: A Benchmark of Plain and Constrained Decoding Generation" — draws a clear line between input and output tasks. For comprehension and retrieval (reading shared context, answering questions about data), TOON's accuracy-per-token ratio is strong. For generation (asking the model to produce structured output), plain JSON had the best one-shot and final accuracy, and constrained-decoding JSON with small structures was even more efficient than TOON.

The mechanism is the prompt tax: when an agent must emit TOON, it needs format-constraining instructions that can run 50–150 tokens. On a short output (a tool-call schema with three to five fields), those instructions cost more than they save. The paper formalizes this as a non-linear scaling hypothesis: TOON's efficiency only pays off once cumulative per-row savings exceed the upfront instruction overhead. For structured agent commands that are typically small and schema-constrained, JSON or native structured output is the right choice.

The practical rule is to split by role. The wire — the shared context and tabular results moving between agents — benefits from TOON. The output schema — the structured commands an agent emits for tool calls or downstream consumers — should stay JSON. See the JSON vs TOON deep-dive for a full breakdown of where each format wins.

The Prompt Tax in a Shared System Prompt

One objection to TOON in multi-agent systems is that each agent needs TOON format instructions, so the prompt tax multiplies. In practice, this is addressable. If agents share a common system prompt that explains the TOON table syntax, and that system prompt is cached, the instruction cost is paid once at write time and then discounted on every read.

Anthropic's prompt caching prices cache reads at 10% of the standard input rate (a 90% discount). OpenAI caches prefixes over 1,024 tokens automatically at roughly 50% of the input rate. A 120-token TOON format instruction block amortizes fast: even at uncached rates, the saving from three hops of a 50-row TOON payload vs JSON exceeds the instruction cost by a factor of ten or more. With caching, the economics are even clearer. For more on stacking these savings, see the guide to optimizing API costs.

TOON in RAG Pipelines: A Natural Fit

Retrieval-augmented generation pipelines are a common multi-agent pattern: a retrieval agent fetches relevant rows from a vector store or database, then one or more reasoning agents consume those results. The retrieved data is almost always tabular — documents with metadata, product records, log entries — which is exactly where TOON's savings are largest.

According to the benchmark, time-series data over 60 days saves 59.0% tokens in TOON versus JSON (9,115 vs 22,245 tokens). GitHub repository data saves 42.3% (8,744 vs 15,144 tokens). These are representative shapes for RAG context. When the retrieval agent serializes its results to TOON before passing them up the pipeline, every downstream agent benefits without any additional conversion cost. For a detailed treatment of this pattern, see optimizing RAG pipelines with TOON.

Model Compatibility Considerations

Not all models perform equally well with TOON. The official benchmark results by model show significant variance: Gemini 3 Flash achieved 96.7% accuracy on TOON, GPT-5 Nano hit 90.9%, while Claude Haiku reached only 59.8% and Grok 4.1 hit 58.4%. In a heterogeneous multi-agent system where different agents run on different models, the weakest model's TOON accuracy becomes the system-level floor.

Before committing to TOON as the universal wire format, test your specific model mix. If a critical agent runs on a smaller model that underperforms on TOON retrieval, either upgrade that agent or fall back to JSON for that hop. The TOON best practices and troubleshooting guide covers model-specific tuning in more detail.

Practical Architecture Pattern

The recommended pattern for a multi-agent system with shared tabular state:

• System prompt (cached): Include TOON format instructions once. On Anthropic, mark this as a cache write; subsequent reads cost 10% of the input rate.
• Retrieval agent output: Serialize fetched rows to TOON before writing to shared state. Use the free converter or the @toon-format/toon npm package to automate this.
• Shared context bus: Pass TOON-encoded tabular data between agents. Each agent reads the same TOON block; the token saving applies at every hop.
• Agent output schema: Agents that emit structured commands, tool-call arguments, or final results use JSON with constrained decoding or native structured output.
• Threshold check: If a shared-state payload is fewer than 10 objects, the prompt-tax math may not work out. Stay with JSON for small one-off payloads.

Frequently Asked Questions