Choosing TOON Delimiters: Comma, Tab, or Pipe?

Use comma (the default) for most datasets. Switch to tab when your values frequently contain commas — it eliminates quoting overhead. Use pipe when neither commas nor pipes appear in your data and you want the most human-readable output. The choice affects quoting and readability; token savings come from TOON's structure, not the delimiter itself.

How TOON's Tabular Block Works

TOON uses a YAML-style indentation model for nesting and a CSV-style tabular block for uniform arrays of objects. Every tabular block opens with a header line that declares the array name, element count, and field list. According to the official TOON reference implementation, the header takes the form name[count]{field1,field2,...}: — for example, users[3]{id,name,role}: — followed by one row per element.

This header is what gives the LLM an explicit schema and count to validate against. The design trades roughly 5% overhead versus the theoretical minimum in exchange for higher retrieval accuracy, as documented on toonformat.dev. The rows that follow the header use the chosen delimiter — comma, tab, or pipe — to separate field values.

The delimiter appears once per field boundary per row. With OpenAI's o200k_base tokenizer averaging roughly 4 characters per token, a single delimiter character is unlikely to consume a full token on its own. What matters more is whether your values contain the delimiter, because values that do must be quoted — adding two quote characters per affected value on every row.

The Same Dataset with Comma, Tab, and Pipe Delimiters

The examples below use a small product catalog. One product name — Laptop, 15-inch — contains a comma. Watch how the quoting requirement changes across delimiter choices.

Comma (default)

Comma is the default delimiter. It is the most familiar and mirrors standard CSV. When a value contains a comma, it must be quoted.

products[4]{id,name,price,in_stock}:
  P001, "Laptop, 15-inch", 999.00, true
  P002, USB-C Hub, 29.99, true
  P003, Wireless Mouse, 19.99, false
  P004, Mechanical Keyboard, 79.99, true

Note that "Laptop, 15-inch" requires quotes. If many of your product names contain commas — as is common with product variants like "Blue, Large" or "Medium, Slim Fit" — every such value adds two characters of quoting overhead on every row.

Tab

Tab-delimited TOON eliminates quoting for values that contain commas. Because product names rarely contain literal tab characters, the same dataset becomes cleaner:

products[4]{id,name,price,in_stock}:
  P001	Laptop, 15-inch	999.00	true
  P002	USB-C Hub	29.99	true
  P003	Wireless Mouse	19.99	false
  P004	Mechanical Keyboard	79.99	true

The tab character (\t) separates fields. Laptop, 15-inch no longer needs quoting because it contains no tab. The trade-off is that tab-delimited output is harder to read in plain text — tabs are invisible in many editors and terminals, and the columns do not visually align unless the editor expands them.

Pipe

Pipe (|) is a human-readable alternative. It is visually distinct, rarely appears in most data domains, and does not require quoting for comma-containing values:

products[4]{id,name,price,in_stock}:
  P001|Laptop, 15-inch|999.00|true
  P002|USB-C Hub|29.99|true
  P003|Wireless Mouse|19.99|false
  P004|Mechanical Keyboard|79.99|true

Pipe strikes the best balance between readability and quoting avoidance for most product and log data. The character is visually distinctive in diffs and terminal output. Its main risk is data that contains pipe characters — SQL-formatted strings, Markdown tables, or certain log formats where | is used as a separator.

Comma vs Tab vs Pipe: When to Use Each

How Quoting Affects Token Count

Quoting matters beyond readability. When a value must be quoted, two double-quote characters are added. With o200k_base averaging roughly 4 characters per token, two quotes add a small but non-zero token cost per affected value per row. Across a large dataset with many quoted values, this adds up.

Consider a product catalog where 30% of names contain commas — a realistic figure for e-commerce data with size and color variants. On a 500-row dataset with five fields, comma-delimited TOON might require quoting on 150 rows. Tab or pipe-delimited TOON eliminates all of those quotes. The toonformat.dev benchmark measured e-commerce order data saving 33.3% tokens versus JSON (73,126 vs 109,599 tokens). Choosing a delimiter that avoids quoting keeps you closer to that optimum.

The broader context: TOON's structural savings — declaring fields once in the header instead of repeating them on every object — are the dominant factor. The delimiter choice is a secondary optimization, but it is the right one to make before deploying at scale. For more on TOON's design rationale, see the TOON specification guide.

Delimiter Choice and CSV Compatibility

Comma-delimited TOON rows are structurally similar to CSV rows, but the formats are not identical. TOON's header line — products[4]{id,name,price,in_stock}: — has no CSV equivalent. A plain CSV header is just a list of field names; TOON's header also encodes the element count and array name, giving the LLM a structural anchor to validate completeness.

This distinction matters when comparing TOON against CSV for LLM prompts. CSV has zero format-instruction overhead because every model is extensively trained on it. TOON's header adds a small fixed cost but enables accuracy gains: the official benchmark shows TOON outperforming CSV on mixed and nested data structures. The CSV vs TOON comparison covers the full trade-off in detail.

If you are converting existing CSV data to TOON, the free converter handles the transformation and lets you preview the output with any delimiter. You can verify the token reduction directly before committing to a delimiter in your pipeline.

Delimiter Selection in Practice

A concise decision process:

• Start with comma. It is the default and requires no configuration. If your data is clean — numeric IDs, short codes, booleans, enumerated statuses — comma works well and the output is immediately readable.
• Inspect for commas in values. If more than a few percent of your string values contain commas, switch to tab or pipe. A quick grep or a sample conversion in the free converter will tell you quickly.
• Choose pipe over tab for human-reviewed output. If TOON-formatted data will be read by developers in logs, diff views, or review tools, pipe is easier to scan than invisible tab stops.
• Choose tab for upstream TSV data. If your source system already produces tab-separated values, tab-delimited TOON is the natural match and avoids a conversion step.
• Check for pipe in your domain. SQL output, shell pipeline logs, and Markdown table content often contain literal pipe characters. Run a sample through the converter and check for quoting before standardizing on pipe.

For a broader look at what TOON is and how the tabular format fits into the full TOON syntax, see What is TOON?. For production deployment guidance including delimiter handling, see the TOON best practices and troubleshooting guide.

Frequently Asked Questions