Token-Efficient AI Agents: TOON for Tool Calls and MCP

AI agents accumulate context with every turn: the system prompt, every tool schema, every prior result, and the new result all re-enter the window simultaneously. That makes tool results — not prompts — the fastest-growing line on your token bill. Encoding those results as TOON before injecting them into the loop compounds the savings across every subsequent turn.

Why tool results dominate token cost in agent loops

A single-turn LLM call has a predictable shape: system prompt + user message + response. An agent loop is different. Each turn re-sends everything the model needs to maintain continuity: the original system prompt, all prior assistant messages, all prior tool calls, and all prior tool results. By turn 10, the context can be dominated by accumulated tool output.

Tool results are also structurally ideal candidates for TOON compression. They are almost always uniform arrays of objects — search results, database rows, API responses — the exact shape where TOON performs best. According to the official TOON benchmarks (current v4.1 run, 5,856 LLM calls across four models), the flat-only datasets used 58.7% fewer tokens than JSON. Time-series data saved 59.0%. Even nested e-commerce-style orders saved 32.9%.

Overall across all data shapes, TOON used 42.6% fewer tokens than JSON while scoring 72.2% retrieval accuracy vs JSON's 71.4%, which works out to 29.2 accuracy-points per 1,000 tokens compared to JSON's 16.6. The accuracy gap is inside the benchmark's 95% confidence intervals, so read it as parity at a much lower token cost.

JSON tool result vs TOON: a concrete example

Consider a tool that returns a list of open GitHub issues. As JSON, a typical batch of five records looks like this:

[
  {"id": 1042, "title": "Fix rate limiter timeout", "state": "open", "author": "alice", "labels": "bug p1"},
  {"id": 1038, "title": "Add CSV export", "state": "open", "author": "bob", "labels": "feature"},
  {"id": 1031, "title": "Docs: update auth section", "state": "open", "author": "carol", "labels": "docs"},
  {"id": 1027, "title": "Memory leak in stream handler", "state": "open", "author": "alice", "labels": "bug p0"},
  {"id": 1019, "title": "Support YAML config", "state": "open", "author": "dave", "labels": "feature p2"}
]

As TOON, the same five records become:

issues[5]{id,title,state,author,labels}:
  1042,Fix rate limiter timeout,open,alice,bug p1
  1038,Add CSV export,open,bob,feature
  1031,"Docs: update auth section",open,carol,docs
  1027,Memory leak in stream handler,open,alice,bug p0
  1019,Support YAML config,open,dave,feature p2

The header line — issues[5]{id,title,state,author,labels}: — gives the model an explicit schema and row count to validate against. Every redundant key, brace, bracket, and quote from the JSON version is gone; only the title containing a colon keeps its quotes. On this kind of uniform array, token counts drop by roughly half. Note that the labels were joined into a single string in the tool wrapper: table rows hold only primitive values, so an array-valued field would push the encoder back to the longer list form.

You can convert any JSON tool result instantly using the JSON to TOON converter, automate it with the @toon-format/toon npm package in your tool wrapper, or call the TOON API over HTTP when the wrapper is not written in JavaScript.

Token accumulation math across N turns

To see why compounding matters, model a simple loop where one tool fires each turn and returns 2,000 tokens of JSON. By turn 10, the agent has re-sent those results a combined 45 times (turn 2 re-sends result 1, turn 3 re-sends results 1 and 2, and so on). Across all turns of a 10-step loop, the total token cost from that one tool's results accumulates:

JSON cost per result:  2,000 tokens
TOON cost per result:    826 tokens  (58.7% savings on uniform arrays)

Cumulative re-sends in a 10-turn loop:
  Turn 1: 0 prior results
  Turn 2: 1 result re-sent
  ...
  Turn 10: 9 results re-sent
  Total re-sends: 0+1+2+...+9 = 45

Extra tokens from re-injection:
  JSON:  45 × 2,000 = 90,000 tokens
  TOON:  45 × 826   = 37,170 tokens
  Saved: 52,830 tokens over 10 turns, from one tool

At gpt-5.6-terra input pricing ($2.00 per 1M tokens on the OpenAI pricing page, as of October 2026), 52,830 tokens is about $0.11 per 10-turn session. Trivial for one session, but at 10,000 sessions per day that is roughly $1,060 saved daily, or about $386,000 per year, from converting a single tool's output format. Prompt caching shrinks these numbers, because re-sent history is often billed at the cached rate ($0.20 per 1M on gpt-5.6-terra), but TOON still cuts the token count that every cache write and read is billed on. Heavier tools, richer results, or longer loops amplify the math further.

For a broader framework on controlling API spend, see our guide to optimizing LLM API costs.

Where to place the conversion: inside the tool wrapper

The right place to convert is as late as possible before the result enters the prompt — inside your MCP server tool handler or equivalent tool wrapper function. This keeps your database, downstream APIs, and business logic working with normal JSON. Only the string that gets returned to the agent loop is TOON. For ad-hoc conversion inside Claude, ChatGPT or Cursor, the hosted TOON MCP server exposes json_to_toon and toon_to_json tools without writing one.

Here is the pattern for an MCP tool handler in TypeScript:

import { encode } from "@toon-format/toon";

// MCP tool handler
async function handleListIssues(args: { repo: string; state: string }) {
  // Fetch raw data — stays as JSON throughout your app
  const issues = await github.issues.list(args);

  // Convert to TOON only at the boundary, before returning to the agent
  const toonResult = encode(issues);

  return {
    content: [{ type: "text", text: toonResult }],
  };
}

MCP does not prescribe a wire format for tool-result content beyond UTF-8 text, so returning a TOON string is fully spec-compliant. The model receives it as plain text and reads it correctly — TOON is a human-readable format that frontier models handle without additional instruction on large, uniform payloads.

If you are building RAG pipelines rather than tool-calling agents, the same principle applies. See optimizing RAG pipelines with TOON for retrieval-specific patterns.

The critical caveat: read TOON, generate JSON

TOON's efficiency advantage is asymmetric. It excels when the model reads structured data from context. It does not excel when the model writes structured data as output.

An independent peer-reviewed benchmark — arXiv 2603.03306, "Token-Oriented Object Notation vs JSON: A Benchmark of Plain and Constrained Decoding Generation" — found that for generation tasks (asking the model to produce structured output), plain JSON had better one-shot and final accuracy. The reason is what the authors call the prompt tax: the instructional overhead required to teach or constrain the model to emit TOON eats into the format savings, especially in short contexts.

The same study identified a scaling threshold: TOON's efficiency is non-linear. On small payloads, the format instructions can cost more than they save. TOON pays off on large, repetitive arrays where the per-row syntax savings amortize the upfront overhead — exactly the shape of most tool results.

The practical recommendation is therefore:

• Tool results injected into context — encode as TOON before returning.
• Tool-call argument schemas (what the model must emit to call the next tool) — keep as JSON. Do not ask the model to generate TOON.
• Model final output — JSON, or whatever your application expects. Do not add a TOON generation requirement.

For a deeper look at how this interacts with model-specific accuracy, the TOON benchmarks post breaks down per-model benchmark results in detail.

Where TOON helps in an agent loop

Not every part of the agent context benefits equally. This table summarizes the verdict for each layer:

The takeaway is that tool results are the only layer that is both large and safe to encode as TOON. Everything else either is not structured data, needs JSON for accurate generation, or is too small to amortize the format overhead.

Caveats and when not to use TOON

TOON's gains are real but not universal. Keep these constraints in mind:

• Small payloads below the scaling threshold. The arXiv study found TOON's efficiency is non-linear: on small, non-repetitive results, the format's upfront overhead can exceed the per-row savings. If your tool returns fewer than 5–10 rows, measure before committing.
• Highly non-uniform data. Flat, uniform arrays are where TOON saves 58–59%. On the official benchmark's mixed-structure track, savings drop to 32.7% vs JSON, and TOON is about 1.6% larger than compact (minified) JSON there. For deeply nested or irregular objects, compact JSON may be the better fit.
• Models with lower TOON comprehension. Per the official benchmarks, accuracy varies widely by model even though TOON roughly ties JSON on each one: grok-4.5 scored 97.1% (TOON) vs 96.3% (JSON), while gpt-5.4-nano scored 57.0% vs 57.4%. If your agent uses a smaller model, test accuracy on your own tasks before deploying TOON in production.
• Long multi-turn agent loops. An independent study, arXiv 2605.29676 ("Notation Matters", 2026), tested token-optimized formats on agentic benchmarks with five open-weight models. TOON saved up to 18% of tokens at a cost of about 9 accuracy points, and the authors conclude that TOON "is not safe as a default in multi-turn agentic systems." Treat TOON in agent loops as an optimization to A/B test, not a default.
• Streaming partial results. If your tool streams tokens mid-result, the TOON table header must be emitted first. Ensure your encoding library flushes the header before streaming rows.

For a broader look at trade-off patterns across chatbot and conversational agents, see building a cost-efficient chatbot with TOON.

Frequently Asked Questions