Halving Costs Twice: The OpenAI Batch API Plus TOON

To cut OpenAI batch costs with TOON, combine two independent discounts: the Batch API takes 50% off both input and output tokens for async jobs, and TOON removes about 43% of tokens on average (42.6% in the official TOON v4.1 benchmark) — up to 58.7% on flat tables. Because fewer tokens enter the 50%-off pricing tier, the reductions multiply rather than add.

What the OpenAI Batch API Actually Costs

The OpenAI Batch API is designed for work that does not need an immediate response: bulk document classification, large-scale extraction, embedding generation, offline report creation. You submit a JSONL file of requests, and the API processes them asynchronously with a 24-hour turnaround window — though according to OpenAI pricing documentation, most batches complete within 1 to 6 hours.

In exchange for that latency tolerance, you receive a 50% discount on both input and output tokens. The batch pool also has a separate, higher rate-limit allocation than the standard synchronous API, which matters for high-volume jobs. There is no special model: the same GPT-5, GPT-4o, and other models are available; you simply route calls through the batch endpoint.

The discount also stacks with prompt caching. For GPT-5.4, a cached batch input can reach $0.625 per million tokens — roughly 75% off the $2.50 standard input rate. TOON sits on top of both: it reduces the token count before any pricing tier is applied.

Why Bulk Jobs Are Exactly Where TOON Saves the Most

The official toonformat.dev benchmarks (v4.1) ran 5,856 LLM calls across 244 questions, 13 datasets, six formats, and four models (claude-haiku-4-5, gemini-3.6-flash, gpt-5.4-nano, grok-4.5). TOON achieved 42.6% fewer tokens than JSON overall, with 58.7% fewer on the flat-only track (68,030 vs 164,643 tokens). Retrieval accuracy was effectively tied: 72.2% vs JSON's 71.4%, within overlapping confidence intervals, which works out to 29.2 accuracy-points per thousand tokens versus JSON's 16.6.

Bulk classification and extraction jobs almost always involve arrays of similar objects — rows from a database, records from a CSV export, product entries, support tickets. That is precisely the data shape where TOON's per-row savings are largest. Each row in a JSON array repeats every key name and every structural glyph. TOON declares fields once in the header and reduces each row to values and delimiters. The larger the batch, the more aggressively the savings compound.

For background on the token-level mechanics, see why JSON is wasteful for LLM prompts and the API cost optimization guide.

Bulk Classification Input: JSON Array vs TOON Table

Consider a batch classification job: label 500 support tickets by category and urgency. Each request in the JSONL batch file contains a small context block plus the ticket data. Here is the data portion of a single batch request, shown as JSON and as TOON:

// ── JSON format — per-request data block (~52 tokens for 5 tickets) ──────
[
  {"ticket_id": "T1001", "subject": "Login broken",        "body": "Cannot sign in since update.", "priority": "high"},
  {"ticket_id": "T1002", "subject": "Slow dashboard",      "body": "Reports take 30s to load.",    "priority": "medium"},
  {"ticket_id": "T1003", "subject": "Export fails",        "body": "CSV export returns 500.",      "priority": "high"},
  {"ticket_id": "T1004", "subject": "Wrong invoice total", "body": "Charged $50 extra this month.","priority": "medium"},
  {"ticket_id": "T1005", "subject": "Feature request",     "body": "Add dark mode please.",        "priority": "low"}
]

// ── TOON format — same data (~27 tokens, ~48% reduction on this slice) ──
tickets[5]{ticket_id,subject,body,priority}:
  T1001, Login broken,        Cannot sign in since update.,  high
  T1002, Slow dashboard,      Reports take 30s to load.,     medium
  T1003, Export fails,        CSV export returns 500.,        high
  T1004, Wrong invoice total, Charged $50 extra this month., medium
  T1005, Feature request,     Add dark mode please.,          low

Across 500 tickets batched in groups of 5, the difference in input tokens is substantial. At batch pricing, you are paying for fewer tokens at a lower rate. If you also place the classification instructions and schema in a cached prefix (which is identical on every call in the batch), the cache discount applies to that portion too.

Standard Sync vs Batch vs Batch Plus TOON: Cost Comparison

The table below shows the relative effect of each strategy on a data-heavy workload. Token figures are illustrative — actual counts depend on your data and model. Pricing is based on cited OpenAI rates. The baseline is a synchronous JSON request at standard input pricing.

The bottom row — batch API, TOON-encoded flat table, with a cached static prefix — represents close to the current practical floor for input costs on a data-heavy repetitive workload. Output token savings also apply: if your extraction response is a structured list matching the same schema, TOON on the output side (used carefully) can further reduce output token count, though for generation tasks JSON remains more reliable per the 2026 arXiv study on TOON vs JSON generation accuracy.

Implementation Checklist for Batch Plus TOON

Before running a batch job with TOON-encoded inputs, verify the following:

• Data shape. TOON's savings are largest on uniform arrays. If your batch items vary significantly in schema, the reduction will be closer to 15–33% than 59%. Check the data shape, for example by pasting one request payload into the token calculator, before committing to the integration.
• Minimum row count per request. The TOON format header (the array[n]{fields}: line) costs a fixed number of tokens. For very small per-request payloads (fewer than roughly 10 rows), the overhead may erode savings. Batch multiple records per request where the task allows it.
• Model accuracy. The official benchmarks show a large variance by model: grok-4.5 at 97.1% vs gpt-5.4-nano at 57.0% on TOON inputs (each model scored about the same on JSON). If you are using a model that scores lower, validate accuracy on a sample before processing the full batch.
• Format instructions in the system prompt. Include a brief TOON format explanation in the system prompt (which you can place in the cached prefix). This is the one-time overhead that amortizes across the entire batch.
• Output format. Request JSON output, not TOON output. The arXiv 2603.03306 study found that plain JSON outperforms TOON on generation tasks. Use TOON for input, JSON or structured outputs for the response.

The sibling post Stacking TOON with Prompt Caching covers the caching mechanics in more detail, including Anthropic's 90% cache-read discount and how to structure the stable prefix.

When the Batch API Is Not the Right Tool

The 50% discount comes with a hard constraint: asynchronous delivery with up to a 24-hour window. This rules out any latency-sensitive use case. Do not use the Batch API for:

• Real-time user-facing responses (chatbots, copilots, search augmentation)
• Workflows that require each result before submitting the next request (sequential pipelines)
• Jobs where data freshness matters more than cost (live dashboards, event-driven triggers)

For those workloads, the combination of TOON encoding and prompt caching alone still delivers meaningful savings without the latency trade-off. See the guide to cost-efficient LLM chatbots for the real-time pattern.

Frequently Asked Questions