Using TOON with Claude: Token Savings and Prompt Caching on the Anthropic API

To use TOON with Claude, encode your retrieved context as TOON in the input messages, place it in a cacheable prefix, and keep any structured output schema as JSON. TOON cuts input tokens by up to 39.9%; Anthropic cache reads cost 10% of the base input rate — the two discounts multiply. But test accuracy on your specific Claude model first: per-model variance is large.

Why Per-Model Accuracy Matters More with Claude

The official TOON benchmark ran 5,016 LLM calls across 209 questions, six formats, and four models. The headline result — 76.4% overall retrieval accuracy and 39.9% fewer tokens versus JSON — is an average. Underneath it, per-model accuracy on TOON varies dramatically: Gemini 3 Flash hit 96.7%, GPT-5 Nano reached 90.9%, and Claude Haiku scored 59.8% — the lowest of the four models tested.

That 37-percentage-point gap between top and bottom is not a flaw in TOON; it reflects how thoroughly each model has internalized the format from training data. Larger Claude models are not represented in the benchmark, so their accuracy is unknown from this source. Before committing TOON to a production Claude pipeline, run a representative sample of your actual questions against your actual Claude model and measure accuracy yourself.

The good news: even if TOON accuracy on your Claude model is lower than ideal for general retrieval, the 99.6% field-retrieval accuracy documented in the benchmark holds for direct lookups — questions of the form "what is the value of field X for record Y." TOON is most reliable when Claude is doing precise field extraction rather than aggregation or filtering.

What a TOON Context Block Looks Like in a Claude Messages Call

Below is the same API call structured two ways — first with JSON context, then with TOON placed in a cacheable prefix. The TOON version is shorter and eligible for Anthropic's cache discount on every repeat call.

// --- JSON context (verbose; keys repeat per row) ---
{
  "model": "claude-sonnet-4-5",
  "system": "You are a data analyst. Answer based only on the provided records.",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Sales records:\n[  {\"region\":\"North\",\"rep\":\"Alice\",\"q1\":42000,\"q2\":38500},  {\"region\":\"South\",\"rep\":\"Bob\",  \"q1\":31000,\"q2\":44000},  {\"region\":\"East\", \"rep\":\"Carol\",\"q1\":55000,\"q2\":51000},  {\"region\":\"West\", \"rep\":\"Dave\", \"q1\":29000,\"q2\":33000}]\nWhich rep had the highest Q2?"
        }
      ]
    }
  ]
}

// --- TOON context with cache_control on the prefix ---
{
  "model": "claude-sonnet-4-5",
  "system": [
    {
      "type": "text",
      "text": "You are a data analyst. Answer based only on the provided records. Data is TOON format: header declares field names and count; rows are comma-separated values.",
      "cache_control": { "type": "ephemeral" }
    }
  ],
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Sales records:\n\nsales[4]{region,rep,q1,q2}:\n  North, Alice, 42000, 38500\n  South, Bob,   31000, 44000\n  East,  Carol, 55000, 51000\n  West,  Dave,  29000, 33000\n\nWhich rep had the highest Q2?",
          "cache_control": { "type": "ephemeral" }
        }
      ]
    }
  ]
}

The cache_control breakpoint tells Anthropic's API to treat everything up to that point as a cacheable prefix. On the first call, the prefix is written to cache at 1.25x the base input rate (5-minute TTL) or 2x (1-hour TTL). On every subsequent call with the same prefix, it is re-read at 10% of the base rate — a 90% discount. Because the TOON block is already up to 39.9% smaller than the equivalent JSON, the cache-write cost is lower and the per-read savings are greater.

Claude Usage Scenarios: Which Format and Caching Approach to Use

The right strategy depends on the shape of your data, the size of your payload, and whether Claude is reading or writing structured content.

How Anthropic Prompt Caching Works and Why Token Count Matters

Anthropic's prompt caching is opt-in: you add a cache_control object with {"type": "ephemeral"} to mark the end of the prefix you want cached. The pricing is asymmetric by TTL:

• Cache write (5-minute TTL): 1.25x the base input rate
• Cache write (1-hour TTL): 2x the base input rate
• Cache read: 10% of the base input rate (a 90% discount)

The break-even point for a 5-minute cache is low: if a call is repeated more than once within the TTL window, the cache reads pay back the write premium. For a 1-hour cache the break-even is around two to three calls, depending on prefix length.

TOON changes the economics on both sides. On the write side, a 39.9% smaller context block means a 39.9% smaller write cost. On the read side, every subsequent cache hit also costs 10% of a smaller base. For workloads that repeat the same context frequently — RAG pipelines, chatbots with a shared knowledge base, agents that always start with the same tool context — the compounding is significant.

One additional detail: tool definitions count toward the prefix. Anthropic adds roughly 313–346 tokens of system overhead per request when tools are supplied. If your workflow uses tool calling, place your TOON data block before the tool definitions so it sits inside the stable, cacheable portion of the prefix. Setting tool_choice to any rather than auto saves approximately 33 additional tokens per call.

For a broader treatment of caching strategy across providers, see our API cost optimization guide.

This Guide vs. the Claude Efficiency Post

The earlier Claude and TOON efficiency guide covers the theoretical token savings and benchmark data in depth. This guide is focused on the practical API integration: how to structure messages, where to place cache_control breakpoints, how to handle tool overhead, and when to fall back to JSON. Read the efficiency post first for the numbers; come back here for the implementation pattern.

The sibling post Using TOON with GPT-5 and the OpenAI API covers the same integration pattern for OpenAI's API, where caching is automatic rather than opt-in and GPT-5 Nano's 90.9% TOON accuracy is stronger than Claude Haiku's 59.8%.

Practical Integration Steps for the Anthropic SDK

Step 1 — Convert your context data to TOON. Use the free json2toon.co converter for one-off testing, or the @toon-format/toon npm package for programmatic conversion. Target uniform arrays — database result sets, retrieved document chunks, structured log batches.

Step 2 — Add a brief format note to your system prompt. One sentence is enough: "Context data is encoded in TOON format. The header line declares field names and record count; rows are comma-separated values." Mark the system message with cache_control if the instructions are stable across calls.

Step 3 — Place TOON data early in the messages array, before any variable content. Everything up to the last cache_control breakpoint is cacheable. The user's actual question should come after, unmarked, so it can vary per call without busting the cache.

Step 4 — Keep output schemas as JSON. Use tool_use blocks or ask for JSON in the final user message. The arXiv 2603.03306 research is clear: for generation, JSON wins on reliability. Parse failures without enforcement run 8–15%; with constrained decoding they drop below 0.1%.

Step 5 — Measure accuracy on your model before going to production. Claude Haiku's 59.8% in the benchmark is a starting point, not a ceiling or a floor for every task. Field retrieval on well-structured TOON data performs far better than aggregation. Run your own evaluation on a representative sample. See the TOON best practices guide for evaluation tips and edge-case handling.

Frequently Asked Questions