Stacking TOON with Prompt Caching to Cut LLM Costs Further

Yes, you can combine TOON with prompt caching — and the savings multiply. Caching discounts the price per token (Anthropic: 90% off on reads; OpenAI: ~50% off cached prefixes). TOON cuts the number of tokens by up to 39.9% overall and 58.8% on flat tables. Stack them by encoding your static context in TOON and placing it in the cached prefix.

Why These Two Cost Levers Are Independent

Most developers treat prompt caching and format optimization as separate concerns. They are not competing strategies — they operate on different dimensions of the cost equation. Your total input cost is:

cost = token_count × price_per_token

Prompt caching attacks price_per_token. According to the Anthropic prompt caching documentation, cache reads cost 10% of the standard input rate — a 90% discount. Cache writes cost 1.25x the base input rate for a 5-minute TTL or 2x for a 1-hour TTL. On OpenAI, caching activates automatically once a stable prefix exceeds 1,024 tokens and is billed at roughly 50% of the input rate.

TOON attacks token_count. The official toonformat.dev benchmarks — 5,016 LLM calls across 209 questions, six formats, and four models — found that TOON uses 39.9% fewer tokens than JSON overall. On flat uniform tables the reduction reaches 58.8% (67,778 vs 164,452 tokens). That is not a niche result: it held across Claude Haiku, Gemini 3 Flash, GPT-5 Nano, and Grok 4.1.

When you apply both levers to the same prefix block, the reduction compounds. A block that starts at 10,000 tokens becomes roughly 6,000 tokens after TOON encoding. Those 6,000 tokens are then cached and re-read at 10% or 50% of the normal rate, depending on the provider. Neither optimization cannibalizes the other.

How to Structure a Prompt for Maximum Cache + TOON Benefit

Prompt caching requires a stable prefix — content that stays identical across requests. This makes it a natural fit for static reference data: product catalogs, user permission tables, knowledge-base excerpts, configuration matrices. That kind of data is also exactly where TOON achieves its largest token savings.

The architecture is straightforward: put your static, TOON-encoded data in the cached portion of the prompt, and inject only the dynamic user question or task at the end.

// ── CACHED PREFIX (sent once, re-read cheaply on every call) ─────────────

// JSON version — 47 tokens for the data portion alone
[
  {"product_id": "P001", "name": "Widget A", "price": 9.99,  "in_stock": true},
  {"product_id": "P002", "name": "Widget B", "price": 14.99, "in_stock": false},
  {"product_id": "P003", "name": "Gadget X", "price": 49.99, "in_stock": true},
  {"product_id": "P004", "name": "Gadget Y", "price": 39.99, "in_stock": true},
  {"product_id": "P005", "name": "Part Z",   "price": 4.99,  "in_stock": false}
]

// TOON version — ~24 tokens for the same data (~49% reduction on this slice)
// Keys declared once; per-row cost is values + commas only
products[5]{product_id,name,price,in_stock}:
  P001, Widget A, 9.99,  true
  P002, Widget B, 14.99, false
  P003, Gadget X, 49.99, true
  P004, Gadget Y, 39.99, true
  P005, Part Z,   4.99,  false

// ── DYNAMIC SUFFIX (injected fresh on every call) ────────────────────────
// "Which products are in stock and under $15?"

Scale this up to a 500-row product catalog and the TOON version might be 4,800 tokens instead of 9,600. On Anthropic, once cached, you re-read those 4,800 tokens at 10% of the input rate instead of paying full price for 9,600 tokens. The write cost is a one-time overhead that amortizes across every subsequent call that hits the cache.

For a broader look at how TOON performs across different use cases, see the API cost optimization guide and the case for replacing JSON in LLM prompts.

Caching Only vs TOON Only vs Both: Relative Cost Effect

The table below uses the cited provider rates to show the qualitative impact of each strategy. Dollar figures are illustrative — actual costs depend on model, tier, and cache hit rate. All figures assume a large static prefix re-read many times (a typical assistant or RAG pattern).

The "Both" rows at the bottom are multiplicative: you are not choosing between strategies, you are applying them to different dimensions of the same cost formula. Note that cache write costs are excluded from this table — they are a one-time overhead per TTL window and typically small relative to the cumulative read savings across many calls.

Does Caching Also Stack with the Batch API?

Yes. On OpenAI, the Batch API's 50% discount on input and output tokens for asynchronous jobs stacks with prompt caching. According to OpenAI pricing, GPT-5.4 cached batch input can reach $0.625 per million tokens — roughly 75% off the $2.50 standard input rate. Apply TOON on top to cut the token count by ~40–59%, and the effective cost per equivalent data payload drops to a fraction of the baseline.

For bulk classification, extraction, or embedding jobs, this three-way combination — TOON encoding, Batch API, and cached prefix — represents the current practical ceiling for LLM cost reduction on data-heavy workloads. The sibling post Halving Costs Twice: The OpenAI Batch API Plus TOON covers the batch side in detail.

Practical Considerations Before You Implement

A few caveats to keep in mind:

• Cache invalidation on data change. If your static context changes, the cache is invalidated and you pay write cost again. Keep truly mutable data out of the cached prefix.
• TOON prompt tax on first call. If the model has not been primed with TOON format instructions, include them in the cached prefix itself — the instructions are static and benefit from caching too. A 2026 arXiv study (2603.03306) found that TOON's format-instruction overhead is a one-time cost that amortizes well on large repeated payloads.
• Minimum prefix size. OpenAI requires a stable prefix of at least 1,024 tokens before caching kicks in. A tiny TOON block might fall below this threshold; in that case, batch the format instructions together with the data to push over the limit.
• Model accuracy variance. The official TOON benchmarks show wide accuracy variance by model: Gemini 3 Flash at 96.7% vs Claude Haiku at 59.8%. Validate TOON accuracy on your specific model and query types before committing to a cached TOON prefix in production.

For a full breakdown of which workloads benefit most from TOON, see the guide to building cost-efficient LLM chatbots.

Frequently Asked Questions