Compressing Chat History with TOON for Cheaper, Longer Conversations

TOON-encode your messages array. Conversation history is a uniform array of role/content objects — the exact shape where TOON saves the most tokens. Official benchmarks show up to 58.8% fewer tokens on uniform arrays versus JSON. Pair that with prompt caching and the two discounts multiply: fewer tokens in the cache block, cheaper to write, cheaper to re-read on every subsequent turn.

Why Conversation History Is an Expensive Line Item

Every LLM API call includes the full conversation history so the model can maintain context. That means you pay for every prior message on every single turn. A 20-turn conversation does not cost 20 units of tokens — it costs roughly 1 + 2 + 3 + … + 20 = 210 units of message-sized chunks. The history grows quadratically relative to cost.

Most developers think about this as a context-window problem — "will the history fit?" — but it is equally a cost problem. Even when the history comfortably fits inside a 128K or 200K context window, you are re-paying for every older message on every new request. For a production chatbot handling thousands of sessions, this compounds quickly. The guide to building cost-efficient chatbots covers the full cost model.

The structure of a messages array is also almost perfectly suited for TOON compression. Every message is an object with the same two or three fields: role, content, and sometimes a name or timestamp. That uniform schema is exactly what TOON's header-once table block was designed to exploit.

JSON vs TOON for a Messages Array: A Direct Comparison

Here is a six-turn conversation encoded both ways. The JSON version repeats the field names on every object. The TOON version declares them once in the header.

// JSON messages array — field names repeat on every object
[
  {"role": "system",    "content": "You are a helpful assistant."},
  {"role": "user",      "content": "What is the capital of France?"},
  {"role": "assistant", "content": "The capital of France is Paris."},
  {"role": "user",      "content": "What is its population?"},
  {"role": "assistant", "content": "Paris has roughly 2.1 million residents in the city proper."},
  {"role": "user",      "content": "Thanks, that is helpful."}
]

// TOON messages table — fields declared once; rows are values only
messages[6]{role,content}:
  system,    You are a helpful assistant.
  user,      What is the capital of France?
  assistant, The capital of France is Paris.
  user,      What is its population?
  assistant, Paris has roughly 2.1 million residents in the city proper.
  user,      Thanks, that is helpful.

The JSON version encodes "role" and "content" as quoted key strings six times each, plus twelve pairs of braces and twelve colons. In the TOON version those field names appear exactly once. As the conversation grows from 6 turns to 60, the JSON overhead scales linearly while the TOON header cost stays constant.

According to the official toonformat.dev benchmarks — which ran 5,016 LLM calls across 209 questions, six formats, and four models — flat uniform arrays see 58.8% fewer tokens in TOON versus JSON (67,778 vs 164,452 tokens). A messages array is exactly a flat uniform array. Overall across all data shapes, TOON achieves 39.9% fewer tokens while maintaining 76.4% retrieval accuracy, slightly above JSON's 75.0%.

How Prompt Caching Stacks with TOON History Compression

Token compression and price-per-token caching are independent discounts that multiply. TOON reduces how many tokens you send; prompt caching reduces what you pay per token for the stable prefix you send repeatedly.

Anthropic: Cache reads cost 10% of the input rate — a 90% discount. Cache writes cost 1.25x or 2x the base input rate depending on TTL (5-minute or 1-hour). The discounts are documented on the Anthropic prompt caching page.

OpenAI: Caching is automatic once a stable prefix exceeds 1,024 tokens. The cached portion is billed at roughly 50% of the input rate. This is ideal for a TOON-encoded history block: you encode the older turns into a compact, stable TOON prefix, and the cache kicks in automatically once that prefix exceeds the threshold. Details are in the prompt caching reference.

The key insight from the benchmarks: caching discounts the price per token; TOON cuts the number of tokens. A smaller cached block is cheaper to write to cache AND cheaper to re-read on every subsequent turn. See the full API cost optimization guide for worked examples with real pricing figures.

Cost and Token Effect: Three Approaches Compared

The table below compares the relative effect of each strategy. Exact totals depend on message length and provider pricing; the directional relationships are grounded in the benchmark and caching data cited above.

For a head-to-head on how TOON compares to raw JSON across different data shapes, see the JSON vs TOON deep-dive.

The Practical Pattern: Compress Old History, Keep Recent Turns Verbatim

Not all turns should be treated the same. The most recent two or three exchanges carry the highest contextual weight — the model needs full fidelity on the immediate back-and-forth. Compressing them into a TOON row truncates nothing, but it does add a tiny parsing step that is worth avoiding for turns the model will attend to most closely.

The recommended pattern is a split approach:

1. Older history (turn N and earlier): encode as a TOON table block. This section is stable and does not change between turns — making it a perfect prompt-caching candidate. Mark it as the cached prefix.
2. Recent context (last 2–3 turns): keep as standard JSON or plain text. This is the active, changing portion of the conversation.
3. New user message: append after the cached prefix in the normal format.

This structure also aligns well with how context window management libraries like LangChain handle message windowing. The older, compressed portion can be summarized further if needed, but TOON compression alone — at up to 58.8% reduction — often buys enough runway before summarization is required.

The arXiv study on TOON (arXiv 2603.03306) confirmed that TOON's efficiency advantage is strongest on large, repetitive payloads — exactly what a long conversation history is. For short conversations of fewer than roughly ten turns, the format-instruction overhead can narrow the gains, so measure on your actual data before committing the pattern to production.

Which Models Handle TOON-Encoded History Best?

The official benchmarks tested four models. Gemini 3 Flash achieved 96.7% overall accuracy with TOON, GPT-5 Nano reached 90.9%, while Claude Haiku scored 59.8% and Grok 4.1 scored 58.4%. For a messages table with only two fields — role and content — every model should perform at or near the top end of its range, since this is one of the simplest possible TOON structures.

Field retrieval accuracy across all models sits at 99.6% — essentially perfect. The model will reliably read each role and content value from the table. The lower accuracy figures appear on aggregation and filtering tasks, which are not relevant to reading conversation history.

Include a single sentence in your system prompt on first use: something like "Conversation history is provided in TOON tabular format: each row is a turn, columns are role and content." After that, no further format instructions are needed within the session, keeping the prompt-tax overhead to a minimum.

For a broader look at how TOON fits into a cost-efficient LLM architecture, see what is TOON? and the companion post on using TOON for log analysis, which covers the same header-once table approach applied to application log data.

Frequently Asked Questions