Using TOON with GPT-5 and the OpenAI API

Feed TOON-encoded data in the input context of your OpenAI API call, keep JSON for the model's output, and let prompt caching stack on top. The official TOON benchmark used the GPT-5 o200k_base tokenizer and found 39.9% fewer input tokens overall — directly reducing the cost of every API call.

Why TOON and GPT-5 Work Well Together

The official TOON benchmark was designed around the GPT-5 family: it used the o200k_base tokenizer (the same one GPT-5 uses) and tested GPT-5 Nano as one of its four models. The results are concrete: GPT-5 Nano scored 90.9% retrieval accuracy on TOON — second only to Gemini 3 Flash at 96.7% — while consuming 39.9% fewer tokens overall versus equivalent JSON. On flat uniform tables, the reduction reached 58.8%.

The reason is structural. OpenAI's o200k_base vocabulary contains roughly 200,000 tokens and averages about four characters per token in English. Every repeated JSON glyph — each {, }, "key", and : — costs tokens on every object in an array. TOON declares field names once in a header line and reduces each row to bare values separated by commas. On an array of 200 records with four fields, that per-row savings compounds across every row.

The benchmark also confirms the right split: TOON is a comprehension format, not a generation format. A 2026 arXiv paper (arXiv 2603.03306) found that when models must produce structured output, plain JSON outperforms TOON on one-shot and final accuracy. Use TOON for the data you send in; use JSON (or structured outputs) for the data you expect back.

What Goes in the OpenAI Call: TOON vs JSON Side by Side

Consider a common pattern: you retrieve a batch of records from a database and pass them to GPT-5 to answer a question. Below is the same API call structured two ways — first with the context as JSON, then with the context as TOON.

// --- JSON context (verbose, keys repeated per row) ---
{
  "model": "gpt-5-nano",
  "messages": [
    {
      "role": "system",
      "content": "You are a support analyst. Answer based only on the provided orders."
    },
    {
      "role": "user",
      "content": "Here are the last 5 orders:\n\n[  { \"id\": 1001, \"customer\": \"Alice\", \"status\": \"shipped\", \"amount\": 49.99 },  { \"id\": 1002, \"customer\": \"Bob\",   \"status\": \"pending\", \"amount\": 12.00 },  { \"id\": 1003, \"customer\": \"Carol\", \"status\": \"shipped\", \"amount\": 89.50 },  { \"id\": 1004, \"customer\": \"Dave\",  \"status\": \"failed\",  \"amount\": 5.00  },  { \"id\": 1005, \"customer\": \"Eve\",   \"status\": \"shipped\", \"amount\": 200.00 }]\n\nHow many orders have shipped, and what is the total shipped amount?"
    }
  ]
}

// --- TOON context (keys declared once; rows are just values) ---
{
  "model": "gpt-5-nano",
  "messages": [
    {
      "role": "system",
      "content": "You are a support analyst. Answer based only on the provided orders. Data format: TOON (header declares field names; rows are comma-separated values)."
    },
    {
      "role": "user",
      "content": "Here are the last 5 orders:\n\norders[5]{id,customer,status,amount}:\n  1001, Alice, shipped, 49.99\n  1002, Bob, pending, 12.00\n  1003, Carol, shipped, 89.50\n  1004, Dave, failed, 5.00\n  1005, Eve, shipped, 200.00\n\nHow many orders have shipped, and what is the total shipped amount?"
    }
  ]
}

The TOON block is meaningfully shorter. The header line orders[5]{id,customer,status,amount}: gives GPT-5 an explicit record count and field schema; each subsequent row is just the values. On five rows the savings are modest, but at 200 rows the 58.8% token reduction for flat data translates directly into a lower invoice from OpenAI.

Notice that the structured output side of the call — what you want GPT-5 to return — stays as JSON via response_format or tools. This is the correct split: TOON for reading, JSON for writing.

Which Part of an OpenAI Call Should Use TOON?

Not every token in an API call is the same. The table below maps each part of a typical OpenAI chat.completions or responses call to the recommended format and the rationale.

How OpenAI Prompt Caching Stacks with TOON

OpenAI's prompt caching is automatic: once a stable prefix exceeds 1,024 tokens, OpenAI caches it and bills subsequent hits at roughly 50% of the standard input rate. Stacking TOON with caching is straightforward — no extra API parameters are needed.

The compounding effect is the key insight. Caching discounts the price per token; TOON cuts the number of tokens. If your context block drops from 10,000 tokens (JSON) to 6,010 tokens (TOON, 39.9% reduction), you are:

• Writing a smaller block on the first call — lower cache-write cost
• Re-reading a smaller block on every subsequent call — 50% rate applied to fewer tokens

The practical setup: put your system prompt and the TOON context block together at the start of the messages array and keep them stable across calls. Vary only the user message at the end. OpenAI's automatic detection will pick up the stable prefix once it passes the 1,024-token threshold.

If you are also using the OpenAI Batch API, the discounts extend further: Batch adds a 50% discount on both input and output tokens, stacking on top of cached-prefix pricing. TOON fits naturally into batch workflows because the context is already serialized and static.

For a detailed treatment of caching strategy across providers, see our guide on optimizing API costs with TOON.

Practical Integration Steps

Integrating TOON into an existing OpenAI workflow takes three steps.

Step 1 — Convert your context data to TOON. Use the free json2toon.co converter for one-off conversions, or install the @toon-format/toon npm package for programmatic conversion in your application. Arrays of uniform objects — database result sets, search hits, log entries — are the best candidates.

Step 2 — Add a one-line format note to your system prompt. Something like: "Context data is encoded in TOON format. The header line declares the field names and record count; rows are comma-separated values." This is the prompt tax, kept minimal. On a large payload it is a negligible fraction of total tokens.

Step 3 — Keep the output schema in JSON. Define your response_format or tools using standard JSON Schema. Do not ask GPT-5 to produce TOON output. The arXiv research is clear that generation accuracy is lower for TOON than for JSON, and constrained-decoding JSON approaches 100% parse reliability versus 8–15% failure rate without enforcement.

For production troubleshooting — handling edge cases, delimiter conflicts, and model-specific accuracy dips — see the TOON best practices and troubleshooting guide.

Accuracy Caveats: When to Test Before Committing

GPT-5 Nano's 90.9% TOON accuracy in the benchmark is strong, but the benchmark methodology matters: 209 questions across six formats and four models, using the o200k_base tokenizer. Accuracy varies significantly by task type even within GPT-5. The official benchmark shows field retrieval at 99.6%, structure awareness at 89.0%, aggregation at 61.9%, and filtering at 56.8%.

If your workload is primarily aggregation or complex filtering rather than field retrieval, validate on your own data before committing TOON to production. The JSON vs TOON comparison has a detailed breakdown of where each format wins and loses by task type.

Also note that the benchmark's scaling hypothesis holds: TOON's efficiency is non-linear. It pays off on large, repetitive payloads where the per-row savings compound across many rows. On small payloads the format-instruction overhead can outweigh the savings. See what is TOON for the design rationale behind this trade-off.

Frequently Asked Questions