Using TOON with GPT-5 and the OpenAI API

Feed TOON-encoded data in the input context of your OpenAI API call, keep JSON for the model's output, and let prompt caching stack on top. The current official TOON benchmark, which counts tokens with GPT-5's o200k_base tokenizer, found 42.6% fewer input tokens overall at the same accuracy as JSON. That directly reduces the cost of every API call.

Why TOON and GPT-5 Work Well Together

The official TOON benchmark counts tokens with o200k_base, the tokenizer used by GPT-5 models, and includes gpt-5.4-nano as one of its four models. In the current run (5,856 calls), gpt-5.4-nano scored 57.0% on TOON and 57.4% on JSON. That is a tie within the confidence intervals. TOON delivered it with 42.6% fewer tokens overall, and 58.7% fewer on flat tables. Across all four models, TOON scored 72.2% against JSON's 71.4%. An earlier run (early 2026) reported 90.9% for the older gpt-5-nano on a different question set, so the two numbers should not be compared directly.

The reason is structural. OpenAI's o200k_base vocabulary contains roughly 200,000 tokens and averages about four characters per token in English. Every repeated JSON glyph — each {, }, "key", and : — costs tokens on every object in an array. TOON declares field names once in a header line and reduces each row to bare values separated by commas. On an array of 200 records with four fields, that per-row savings compounds across every row.

The benchmark also confirms the right split: TOON is a comprehension format, not a generation format. A 2026 arXiv paper (arXiv 2603.03306) found that when models must produce structured output, plain JSON outperforms TOON on one-shot and final accuracy. Use TOON for the data you send in; use JSON (or structured outputs) for the data you expect back.

Which GPT-5 Model Should You Pair with TOON?

OpenAI released GPT-5.6 on July 9, 2026, in three tiers: Sol, Terra, and Luna. As of October 2026, the OpenAI API pricing page lists these standard-tier prices per million tokens:

TOON's savings matter most on input-heavy workloads such as RAG, analytics over database rows, and agent tool results, because input tokens are where the data lives. On gpt-5.6-terra, 100M input tokens of JSON context per day costs $200. At the 42.6% overall reduction, the same context as TOON costs about $115. Only gpt-5.4-nano appears in the official benchmark, so run your own accuracy check before moving TOON prompts to Sol, Terra, or Luna.

What Goes in the OpenAI Call: TOON vs JSON Side by Side

Consider a common pattern: you retrieve a batch of records from a database and pass them to GPT-5 to answer a question. Below is the same API call structured two ways — first with the context as JSON, then with the context as TOON.

// --- JSON context (verbose, keys repeated per row) ---
{
  "model": "gpt-5.6-luna",
  "messages": [
    {
      "role": "system",
      "content": "You are a support analyst. Answer based only on the provided orders."
    },
    {
      "role": "user",
      "content": "Here are the last 5 orders:\n\n[  { \"id\": 1001, \"customer\": \"Alice\", \"status\": \"shipped\", \"amount\": 49.99 },  { \"id\": 1002, \"customer\": \"Bob\",   \"status\": \"pending\", \"amount\": 12.00 },  { \"id\": 1003, \"customer\": \"Carol\", \"status\": \"shipped\", \"amount\": 89.50 },  { \"id\": 1004, \"customer\": \"Dave\",  \"status\": \"failed\",  \"amount\": 5.00  },  { \"id\": 1005, \"customer\": \"Eve\",   \"status\": \"shipped\", \"amount\": 200.00 }]\n\nHow many orders have shipped, and what is the total shipped amount?"
    }
  ]
}

// --- TOON context (keys declared once; rows are just values) ---
{
  "model": "gpt-5.6-luna",
  "messages": [
    {
      "role": "system",
      "content": "You are a support analyst. Answer based only on the provided orders. Data format: TOON (header declares field names; rows are comma-separated values)."
    },
    {
      "role": "user",
      "content": "Here are the last 5 orders:\n\norders[5]{id,customer,status,amount}:\n  1001,Alice,shipped,49.99\n  1002,Bob,pending,12\n  1003,Carol,shipped,89.5\n  1004,Dave,failed,5\n  1005,Eve,shipped,200\n\nHow many orders have shipped, and what is the total shipped amount?"
    }
  ]
}

The TOON block is meaningfully shorter. The header line orders[5]{id,customer,status,amount}: gives GPT-5 an explicit record count and field schema; each subsequent row is just the values. On five rows the savings are modest, but at 200 rows the 58.7% token reduction the benchmark measured on flat data translates directly into a lower invoice from OpenAI.

Notice that the structured output side of the call — what you want GPT-5 to return — stays as JSON via response_format or tools. This is the correct split: TOON for reading, JSON for writing.

Which Part of an OpenAI Call Should Use TOON?

Not every token in an API call is the same. The table below maps each part of a typical OpenAI chat.completions or responses call to the recommended format and the rationale.

How OpenAI Prompt Caching Stacks with TOON

OpenAI's prompt caching is automatic: once a stable prefix exceeds 1,024 tokens, OpenAI caches it. As of October 2026, cache hits on GPT-5.x models bill at 10% of the standard input rate (for example $0.20 vs $2.00 per million on gpt-5.6-terra). Stacking TOON with caching is straightforward — no extra API parameters are needed.

The compounding effect is the key insight. Caching discounts the price per token; TOON cuts the number of tokens. If your context block drops from 10,000 tokens (JSON) to about 5,740 tokens (TOON, at the 42.6% overall reduction), you are:

• Writing a smaller block on the first call — lower cache-write cost
• Re-reading a smaller block on every subsequent call, with the 10% cached rate applied to fewer tokens

The practical setup: put your system prompt and the TOON context block together at the start of the messages array and keep them stable across calls. Vary only the user message at the end. OpenAI's automatic detection will pick up the stable prefix once it passes the 1,024-token threshold.

If you are also using the OpenAI Batch API, the discounts extend further: Batch pricing is roughly 50% off standard rates for asynchronous jobs. TOON fits naturally into batch workflows because the context is already serialized and static.

For a detailed treatment of caching strategy across providers, see our guide on optimizing API costs with TOON.

Practical Integration Steps

Integrating TOON into an existing OpenAI workflow takes three steps.

Step 1 — Convert your context data to TOON. Use the free JSON to TOON converter for one-off conversions, or install the @toon-format/toon npm package for programmatic conversion in your application. Arrays of uniform objects — database result sets, search hits, log entries — are the best candidates.

Step 2 — Add a one-line format note to your system prompt. Something like: "Context data is encoded in TOON format. The header line declares the field names and record count; rows are comma-separated values." This is the prompt tax, kept minimal. On a large payload it is a negligible fraction of total tokens.

Step 3 — Keep the output schema in JSON. Define your response_format or tools using standard JSON Schema. Do not ask GPT-5 to produce TOON output. The arXiv research is clear that generation accuracy is lower for TOON than for JSON, and constrained-decoding JSON approaches 100% parse reliability versus 8–15% failure rate without enforcement.

For production troubleshooting — handling edge cases, delimiter conflicts, and model-specific accuracy dips — see the TOON best practices and troubleshooting guide.

Accuracy Caveats: When to Test Before Committing

gpt-5.4-nano's 57.0% on TOON tied with JSON's 57.4%. The methodology matters here: 244 questions across six formats and four models, with reasoning disabled. Accuracy varies a lot by task type. Across all models, the current run shows TOON vs JSON at 97.8% vs 99.2% for field retrieval, 90.3% vs 84.0% for structure awareness, 48.4% vs 48.4% for aggregation, and 38.0% vs 41.1% for filtering.

If your workload is primarily aggregation or complex filtering rather than field retrieval, validate on your own data before committing TOON to production. The JSON vs TOON comparison has a detailed breakdown of where each format wins and loses by task type.

Also note that the benchmark's scaling hypothesis holds: TOON's efficiency is non-linear. It pays off on large, repetitive payloads where the per-row savings compound across many rows. On small payloads the format-instruction overhead can outweigh the savings. See what is TOON for the design rationale behind this trade-off.

Frequently Asked Questions