Building Fine-Tuning Datasets: JSONL Wrappers, TOON Content

Yes — with a clear split. Fine-tuning files stay JSONL and the chat envelope stays JSON; do not change those. But the structured data you embed inside each user message can be TOON, which trims training tokens by up to 58.8% and teaches the model to read TOON format. For target completions the model must generate, keep those in JSON.

The Three-Layer Rule: File Format, Envelope, Content

When people ask whether they can use TOON in fine-tuning datasets, they are often conflating three distinct layers of a training example. Getting this distinction right determines whether you save tokens or introduce subtle accuracy problems.

Layer 1 — File format. Fine-tuning providers (OpenAI, Anthropic, together.ai, and others) require JSONL: one complete JSON object per line, no trailing commas, no wrapper array. This is a parsing and pipeline requirement, not an LLM format question. Do not change it to TOON-L or any other format. The file stays JSONL.

Layer 2 — Chat envelope. Each line contains a messages array with role/content objects, and sometimes a top-level tools or functions key. This structure is what the training framework reads to construct the supervised signal. It must remain valid JSON — the envelope does not change.

Layer 3 — Message content. The string value inside a content field is just text. If you are passing a table of records to the model as part of a user turn, that table can be formatted as TOON. The training framework treats it as an opaque string. The model learns from it during training, just as it would learn from any other text. This is the layer where TOON belongs.

How Much Does TOON Reduce Training Token Count?

Training compute is billed by token. Fewer tokens in training examples means lower cost and faster training runs. The official TOON benchmarks — 5,016 LLM calls across 209 questions, six formats, and four models — quantify the savings by data shape:

• Flat uniform tables: 58.8% fewer tokens than JSON (67,778 vs 164,452 tokens)
• Time-series data: 59.0% fewer tokens (9,115 vs 22,245 tokens)
• GitHub repo / structured records: 42.3% fewer tokens
• E-commerce orders with nesting: 33.3% fewer tokens
• Mixed structures: 21.9% fewer tokens
• Overall average: 39.9% fewer tokens than JSON

If your training examples contain thousands of records embedded in user messages, those percentages translate directly into training cost. A dataset with 50,000 examples where each user message embeds a 20-row table of uniform records could shrink from 164,000 tokens to 68,000 tokens for the content portion alone — nearly 60% less.

The Prompt Tax — and How Fine-Tuning Eliminates It

There is an important caveat to TOON's token efficiency: the prompt tax. At inference time, a base model that has not been trained on TOON requires format instructions — typically 50 to 150 tokens — to tell it how to read the array[n]{field1,field2}: table syntax. Those instructions are overhead that partially offsets TOON's per-row savings on short contexts.

A 2026 arXiv study (arXiv 2603.03306) — "Token-Oriented Object Notation vs JSON: A Benchmark of Plain and Constrained Decoding Generation" — identified this as a scaling threshold: TOON's efficiency is non-linear, paying off only when cumulative per-row savings amortize the upfront instruction overhead. On small payloads, the tax can exceed the savings.

Fine-tuning addresses this directly. When you include TOON-formatted content in training examples, the model sees the format repeatedly across the dataset and internalizes the parsing rules. After fine-tuning, you can reduce or eliminate the per-call TOON format instructions at inference time, because the model already knows the format. The prompt tax is paid once — during training — instead of on every inference call.

This is one of the stronger arguments for embedding TOON in fine-tuning data: you are simultaneously reducing training cost and amortizing the inference-time overhead that would otherwise make TOON impractical on smaller payloads.

What a TOON Fine-Tuning Example Looks Like

Below is a single JSONL training line. The outer structure is standard JSON, the envelope follows the chat format, and the user message content embeds a TOON table. The assistant target is plain JSON — for reasons explained in the next section.

{"messages": [
  {
    "role": "system",
    "content": "You are a data analyst. Answer questions about the provided records."
  },
  {
    "role": "user",
    "content": "Here are this week's order records:\n\norders[5]{order_id,customer,product,qty,total_usd}:\n  ORD-001, alice@example.com, Wireless Keyboard, 2, 99.98\n  ORD-002, bob@example.com, USB-C Hub 7-in-1, 1, 34.99\n  ORD-003, carol@example.com, Webcam 1080p, 3, 209.97\n  ORD-004, alice@example.com, Monitor Stand, 1, 29.99\n  ORD-005, dave@example.com, Laptop Stand, 2, 79.98\n\nWhich customer placed the most orders this week?"
  },
  {
    "role": "assistant",
    "content": "{"customer": "alice@example.com", "order_count": 2}"
  }
]}

The TOON table in the user turn declares five records with five fields in a single header. The five rows contain only values — no repeated field names, no repeated braces or quotes. The model learns from the contrast: it receives TOON for reading, and it outputs clean JSON.

Notice that the assistant target {"customer": "alice@example.com", "order_count": 2} is plain JSON. That choice is deliberate and grounded in research, as the next section explains.

Why Target Completions Should Stay in JSON

The arXiv 2603.03306 study examined not just comprehension but generation: asking the model to produce structured output. Its finding is direct: plain JSON had the best one-shot and final accuracy for generation tasks. TOON's advantage — the accuracy-per-token efficiency — applies to comprehension and retrieval, not to production of structured text.

The reason traces to training data distribution. Every major LLM was pre-trained on vast amounts of JSON — API responses, configuration files, web data. The model's internal representation of "structured output" is deeply shaped by JSON syntax. When you constrain or supervise it to output JSON, you are aligning with that prior. Constraining it to output TOON requires overcoming that prior, and the arXiv data shows this is harder in short-context or one-shot settings.

The practical rule is clean: TOON for input data that the model reads, JSON for output data that the model generates. Your fine-tuning dataset should reflect this split. User message content with embedded structured data uses TOON. Assistant completions that are structured responses use JSON.

This split also aligns with how you will use the fine-tuned model at inference time. For more on the input-versus-output distinction in production, see our guide on when to stop using JSON for LLMs and the JSON vs TOON deep-dive.

Which Part of the Training Example Uses Which Format?

The following table summarizes the format choice for each part of a fine-tuning example, with the rationale drawn from the TOON benchmarks and the arXiv study:

Practical Notes on Dataset Construction

A few implementation details that come up when actually building these datasets:

Schema consistency within a dataset. If you are training on multiple record types, keep the TOON header consistent for each type across all examples. The model learns the mapping between header names and values from repetition. Varying the field order or abbreviations across examples adds noise to the learning signal.

Minimum viable table size. TOON's token savings scale with the number of rows. For very small payloads — two or three records — the header overhead can consume a meaningful fraction of the savings. A rough rule: use TOON in training examples when the embedded table has five or more rows. Below that, JSON is fine and simpler. The arXiv study confirms this threshold behavior: TOON's efficiency is non-linear and pays off at scale.

Converting existing datasets. If you have an existing fine-tuning dataset where user messages embed JSON arrays, converting the data portions to TOON is straightforward using the free json2toon converter. The file format and envelope remain unchanged; only the embedded string content changes.

Validation after conversion. After converting, spot-check that the TOON tables in your training examples parse correctly and that field names are unambiguous. A malformed header in a training example is silent during conversion but creates a confusing training signal. The JSON to TOON migration guide covers common conversion edge cases.

Frequently Asked Questions