TOON, Structured Outputs & Function Calling: Pick the Right Format for Each Direction

Use JSON for what the model writes — function arguments and structured outputs — and TOON for what you feed in — tool results, retrieved rows, and injected context. A 2026 arXiv study confirms JSON wins generation accuracy while TOON wins comprehension efficiency. The direction of data flow determines the right format.

Why Direction Matters More Than Format

Most discussions about TOON vs JSON treat them as global choices: pick one and use it everywhere. That framing misses the core insight from the independent benchmark arXiv 2603.03306 — "Token-Oriented Object Notation vs JSON: A Benchmark of Plain and Constrained Decoding Generation" (February 2026). The paper found that JSON has the best one-shot and final accuracy when a model must produce structured output, while TOON's edge is in comprehension and retrieval, meaning data the model reads rather than data it generates.

Function calling puts both directions in the same loop. The model outputs structured function arguments (generation); you execute the function and return structured data back (comprehension input). Applying a single format to both directions ignores what each format does best.

The official toonformat.dev benchmarks reinforce this: TOON achieved 39.9% fewer tokens overall and up to 58.8% fewer tokens on flat uniform tables, with a retrieval accuracy of 76.4% versus JSON's 75.0%. The wins are on the input side. For a deeper look at that token-savings profile, see our JSON vs TOON comparison.

How Much Does Schema Overhead Actually Cost?

Before optimizing tool results, it is worth understanding how much the schema definition itself costs. According to production data compiled in the Structured Output Production Guide 2026, schema token overhead scales with complexity:

• A 3-field schema costs roughly 50 tokens
• A 20-field schema costs roughly 500 tokens
• A deeply nested schema can reach 2,000 tokens

On top of the schema itself, Anthropic adds approximately 313–346 tokens of system overhead per request whenever any tools are supplied. Using tool_choice: "any" instead of leaving the default saves around 33 of those tokens — a small but free optimization.

This overhead is the fixed cost of structured output. It does not go away regardless of what format you use for tool results. What you can control is the size of every tool result that flows back in. That is where TOON's input savings compound across a multi-turn conversation.

Why Use JSON for Model Output (Function Arguments)?

The reliability case for JSON on the output side is unambiguous. Without format enforcement, JSON parsing fails 8–15% of the time across production workloads. Native structured output with constrained decoding drives that rate below 0.1% — an improvement of two orders of magnitude, based on analysis of approximately 2 million calls. Reliability tiers break down as follows:

• Prompt-engineering only: 80–95% compliance
• Function calling / schema-enforced: 95–99%
• Native structured output (constrained decoding): ~100%

TOON does not have a constrained-decoding implementation in the major inference APIs. Asking the model to output TOON relies on prompt engineering alone, putting you in the 80–95% tier at best. For function arguments that your code will parse and execute, that failure rate is unacceptable in production.

The arXiv paper formalizes this: even without constrained decoding, plain JSON outperformed TOON on generation tasks in one-shot evaluations because models have deeply internalized JSON from training. The "prompt tax" — the instructional overhead required to teach the model to write TOON — can erase TOON's gains entirely on generation. See our TOON best practices guide for how to size the prompt tax for your payload.

A Complete Tool-Calling Loop: JSON Out, TOON In

The following example shows a product-search tool call. The model emits JSON arguments (constrained decoding), your server executes the search, and you return the results as TOON for the next context window. The contrast between the JSON and TOON versions of the same tool result illustrates the input savings directly.

// STEP 1: Model emits function arguments as JSON (constrained decoding).
// No TOON here — JSON's reliability tier is ~100% with enforcement.
{
  "function": "search_products",
  "arguments": {
    "category": "laptops",
    "max_price": 1200,
    "in_stock": true
  }
}

// STEP 2a: Tool result returned as JSON (the old approach).
// 8 products × repeated keys = heavy token cost per turn.
[
  {"id": "LP-001", "name": "ProBook 14",  "price": 999,  "stock": 14, "rating": 4.6},
  {"id": "LP-002", "name": "UltraSlim",  "price": 1149, "stock": 3,  "rating": 4.8},
  {"id": "LP-003", "name": "WorkStation", "price": 1099, "stock": 7,  "rating": 4.4},
  {"id": "LP-004", "name": "DevBook Pro", "price": 1199, "stock": 2,  "rating": 4.7},
  {"id": "LP-005", "name": "NanoEdge",   "price": 849,  "stock": 11, "rating": 4.3},
  {"id": "LP-006", "name": "CoreMax",    "price": 1050, "stock": 5,  "rating": 4.5},
  {"id": "LP-007", "name": "ThinPad X1", "price": 1175, "stock": 9,  "rating": 4.9},
  {"id": "LP-008", "name": "SwiftBook",  "price": 925,  "stock": 22, "rating": 4.2}
]
// ≈ 210 tokens (keys repeated 8×)

// STEP 2b: Same tool result returned as TOON (the optimized approach).
// Fields declared once in the header; rows are pure values.
products[8]{id,name,price,stock,rating}:
  LP-001, ProBook 14,   999,  14, 4.6
  LP-002, UltraSlim,   1149,  3, 4.8
  LP-003, WorkStation, 1099,  7, 4.4
  LP-004, DevBook Pro, 1199,  2, 4.7
  LP-005, NanoEdge,     849, 11, 4.3
  LP-006, CoreMax,     1050,  5, 4.5
  LP-007, ThinPad X1,  1175,  9, 4.9
  LP-008, SwiftBook,    925, 22, 4.2
// ≈ 125 tokens — roughly 40% smaller; savings compound across every multi-turn step

// STEP 3: Model reads TOON context, emits JSON for the next action.
// The format split is invisible to the end user; only the token bill changes.

In a multi-step agent loop where tool results accumulate in the context window, each TOON-encoded result is smaller than its JSON equivalent. Over five or ten turns the savings multiply. For a deeper dive on this pattern in retrieval-augmented pipelines, see optimizing RAG pipelines with TOON.

Direction Table: Which Format for Which Side?

The following table maps each message direction in a tool-calling loop to the recommended format and the primary reason.

Practical Implementation Notes

Applying the split in practice requires two small additions to an existing tool-calling setup. Neither requires changes to the model API calls themselves.

First, on the tool-execution side, serialize the result to TOON before inserting it into the tool-result message. The @toon-format/toon npm package provides a stringify() function that mirrors JSON.stringify(). For a walk-through of the API, see the TOON introduction.

Second, add a brief TOON-format description to your system prompt — one or two sentences covering the header syntax and that values are comma-delimited. This is the prompt tax, and for any tool result larger than roughly ten rows it pays for itself immediately. The TOON best practices guide has copy-paste system-prompt snippets.

If your tool results include deeply nested objects rather than uniform arrays — for example, a raw API response with mixed types and irregular keys — stick with JSON for those results. TOON only saves 21.9% on mixed structures, and the nesting prevents the header optimization from firing. The direction split applies cleanly to tabular results; for non-uniform data, JSON remains the correct input format too.

For production pipelines that need schema validation, sub-millisecond indexed lookups over tool results, or streaming over large datasets, consider upgrading from TOON to TONL, which supports 32–50% token savings along with a built-in SQL-like query API and zero runtime dependencies.

Frequently Asked Questions