Few-Shot Prompting with TOON: Cheaper, Clearer Examples

Use TOON for the input context portion of your few-shot examples and JSON for the demonstrated output. A few-shot block is a uniform array — TOON's strongest case at 39.9% to 58.8% fewer tokens. But research confirms JSON wins for generation accuracy, so show the model what output you want in the format you want it back.

Why Few-Shot Format Matters More Than People Realize

Few-shot prompting works by prepending a set of input-output examples before the actual query. Those examples are repeated verbatim on every single API call. Unlike a one-off context injection, the few-shot block is structural — it lives in your prompt template and compounds across your entire request volume.

If your few-shot block is 800 tokens of JSON and you make 100,000 calls per month, that is 80 million tokens spent purely on format overhead — keys, braces, quotes, and commas that the model does not need to understand your examples. Switching that block to TOON does not change what information the model sees; it changes how efficiently that information is encoded.

The toonformat.dev benchmark ran 5,016 LLM calls across 209 questions, six formats, and four models. Overall, TOON achieved 39.9% fewer tokens than JSON while maintaining 76.4% retrieval accuracy versus JSON's 75.0%. On flat uniform data — exactly what a few-shot example block is — the reduction reached 58.8% (67,778 vs 164,452 tokens). That is not a marginal improvement; it is roughly half the token cost.

For context on how JSON and TOON differ at the tokenization level, see our JSON vs TOON deep-dive. The short version: every repeated key and structural character in JSON is a token. TOON declares field names once in the header and pays only for values on each row.

The Critical Split: TOON for Input, JSON for Output

Before encoding your entire few-shot block as TOON, there is an important caveat from independent research. The paper Token-Oriented Object Notation vs JSON: A Benchmark of Plain and Constrained Decoding Generation (arXiv 2603.03306, February 2026) found that for generation tasks — where you are asking the model to produce structured output — plain JSON had the best one-shot and final accuracy. TOON's advantage is for comprehension and retrieval, not for the model generating a format it needs to write.

Applied to few-shot prompting, this creates a clean split:

• Input context (the data you show the model as the "given" in each example): encode as TOON. The model reads it; 99.6% field retrieval accuracy means it reads it reliably.
• Demonstrated output (what you show the model as the "correct answer" in each example): encode in your target output format. If you want JSON back, show JSON. The model learns the output format by imitating the examples, so the output examples must match what you want it to produce.

This split is also consistent with findings on structured outputs more broadly. The same research area notes that without enforcement, JSON parsing fails 8–15% of the time in production; with native constrained decoding, failure drops below 0.1%. Demonstrating JSON output in your examples supports the model's existing internalized JSON generation behavior — a behavior built up through far more training exposure than TOON has.

Few-Shot Block: JSON vs TOON Side by Side

Consider a sentiment classification task. Each example gives the model a review record and expects a label. Here is what a four-example block looks like in each format.

// FEW-SHOT BLOCK — JSON format (inputs and outputs both JSON)
// Input context repeated as full objects on every example

[
  {
    "input": {"review_id":"R001","product":"Widget Pro","text":"Excellent build quality, arrived fast.","rating":5},
    "output": {"label":"positive","confidence":0.97}
  },
  {
    "input": {"review_id":"R002","product":"Gadget Lite","text":"Stopped working after two weeks.","rating":1},
    "output": {"label":"negative","confidence":0.99}
  },
  {
    "input": {"review_id":"R003","product":"Part X","text":"Decent for the price, nothing special.","rating":3},
    "output": {"label":"neutral","confidence":0.84}
  },
  {
    "input": {"review_id":"R004","product":"Widget Pro","text":"Missing screws in the box, very disappointed.","rating":2},
    "output": {"label":"negative","confidence":0.95}
  }
]
// Approximate token count: ~280 tokens

// FEW-SHOT BLOCK — TOON inputs, JSON outputs
// Input context as a TOON table; demonstrated output stays JSON

reviews[4]{review_id,product,text,rating}:
  R001, Widget Pro, Excellent build quality arrived fast, 5
  R002, Gadget Lite, Stopped working after two weeks, 1
  R003, Part X, Decent for the price nothing special, 3
  R004, Widget Pro, Missing screws in the box very disappointed, 2

// Expected output for each (JSON — matches the format you want back):
R001 -> {"label":"positive","confidence":0.97}
R002 -> {"label":"negative","confidence":0.99}
R003 -> {"label":"neutral","confidence":0.84}
R004 -> {"label":"negative","confidence":0.95}
// Approximate token count: ~165 tokens — roughly 41% fewer

The TOON version compresses the input context while leaving the output demonstration in JSON. The model sees the same information, learns the same output schema, and the block costs significantly fewer tokens per call.

Convert your existing JSON few-shot blocks to TOON instantly with the free json2toon.co converter.

Format Each Part of a Few-Shot Example: Decision Table

Stacking TOON with Prompt Caching

Few-shot blocks are the ideal candidate for prompt caching because they are stable — the same examples appear in every call. TOON makes that cache even more valuable by reducing the size of the block being cached.

Anthropic caches stable prefixes at 10% of the input token rate — a 90% discount on reads — with a 5-minute TTL at 1.25x write cost, or a 1-hour TTL at 2x write cost. (Anthropic prompt caching docs.)

OpenAI applies caching automatically once a stable prefix exceeds 1,024 tokens, billing the cached portion at approximately 50% of the standard input rate. (Prompt caching overview.)

The two discounts multiply. TOON reduces the number of tokens in the block. Prompt caching reduces the price per token of that block on every read. A few-shot prefix that starts at 800 JSON tokens, reduced to 490 TOON tokens, then cached at 50% of input rate, costs approximately 38% of what the original JSON-uncached block cost — a 62% total reduction from two independent optimizations.

For a full breakdown of how TOON and caching interact across different providers, see our guide to optimizing LLM API costs.

When TOON Does Not Help Few-Shot Prompts

Not every few-shot block benefits from TOON. Two cases where you should stick with JSON:

Small example counts. If your few-shot block contains three or fewer examples with simple structures, the TOON header and any required format-instruction tokens can exceed the per-row savings. The arXiv 2603.03306 paper formalizes this as the prompt tax: a non-linear threshold below which format overhead outweighs compression. The TOON best practices guide covers the break-even calculation in detail.

Non-uniform examples. TOON's header declares a fixed set of fields for every row. If your examples vary in shape — one has five fields, the next has eight different fields — the flat-table structure does not map cleanly. In that case, JSON's per-object flexibility is the right choice.

For most production classification, extraction, or summarization pipelines where examples are drawn from a consistent schema, neither of these limitations applies. A review classification block, an entity extraction block, or a code-tagging block all have uniform schemas — the common case favors TOON.

Practical Implementation Steps

Moving an existing few-shot prompt from all-JSON to TOON-input / JSON-output requires three changes:

1. Export your existing few-shot input examples as a JSON array and run them through the json2toon.co converter to get the TOON table.
2. Keep your output demonstrations in JSON exactly as they are — do not convert them.
3. Add a brief format note in your system prompt (one sentence: "Input records are in TOON format: a header line declares fields, each row lists values in order."). This covers the prompt tax and is stable enough to sit in a cached system prompt.

If you are using Anthropic or OpenAI with prompt caching, mark the few-shot prefix as a cacheable block. The format instruction and TOON table together form a stable prefix that qualifies for caching discounts on both platforms.

For a broader view of how TOON compares across different prompt components, see what is TOON — a complete introduction to the format and its trade-offs.

Frequently Asked Questions