TOON Format Specification: Grammar and Syntax Guide

Status: Working Draft, spec version 4.1 (2026-07-26)
This guide summarizes the technical specification for TOON (Token-Oriented Object Notation), a lossless, indentation-based encoding of the JSON data model optimized for Large Language Model (LLM) context windows. The normative text lives in the official SPEC.md (MIT, by Johann Schopplich); see the official project page for updates.

The key goals of this specification are:

• Maximizing Token Density: Representing data in the fewest possible BPE (Byte Pair Encoding) tokens. In the current official benchmark (v4.1, 5,856 LLM calls), TOON used 42.6% fewer tokens than formatted JSON while scoring 72.2% accuracy versus JSON's 71.4%.
• Human Readability: Maintaining a structure that is intuitive for humans and easy for LLMs to reason about.
• Lossless JSON Conversion: Ensuring 1:1 mapping with standard JSON types (objects, arrays, strings, numbers, booleans, null).

The reference implementation is the @toon-format/toon npm package (4.1.1 at the time of writing). Every TOON example on this page was produced by running that encoder. You can reproduce them in the json2toon converter.

What changed in spec 4.0 and 4.1?

Spec 4.0 (July 2026) and 4.1 (July 26, 2026) are the biggest revisions since TOON launched, according to the spec changelog:

• Added in 4.0: full-line # comments, keyed tabular form ([N:]), nested field groups (customer{name,country} inside a header), and a normative number grammar.
• Removed in 4.0: key folding and path expansion (keyFolding, flattenDepth, expandPaths). The 4.x library ignores these options.
• Stricter in 4.0: indentation depth jumps are an error in strict mode, and strings starting with # must be quoted.
• 4.1: encoders lost latitude. Tabular form is mandatory whenever detection succeeds, empty arrays are always key: [], a leading BOM is stripped, and the indentation option is named indentSize.

The changelog advises upgrading decoders before encoders, because older decoders cannot read keyed tables or nested field groups. The TOON v4 migration checklist walks through the upgrade.

1. Document Structure

A TOON document MUST be encoded in UTF-8. The file extension is .toon and the provisional media type is text/toon. A TOON document represents exactly one root value: an object, an array, or a single primitive.

2. Whitespace and Indentation

TOON uses Significant Whitespace to denote hierarchy.

• Indent Unit: The default indentation is 2 spaces (U+0020), configurable with the encoder's indentSize option. Tabs (U+0009) MUST NOT be used for indentation. A tab is still a valid delimiter (see section 4.3).
• Depth jumps: Each nested line must be exactly one level deeper than its parent. In strict mode (the default), skipping a level is an error: a child of a: indented by 4 spaces instead of 2 fails with Indentation depth jump: expected depth 1, but found 2.
• Newlines: Encoders emit LF (\n) line endings with no trailing spaces or trailing newline.

3. Objects

An Object is a collection of key-value pairs.

Syntax: key: value

user:
  name: Alice
  role: admin

• The colon is mandatory. A single space follows it when a primitive value is on the same line.
• If the value is a nested object, the line ends right after the colon and the children are indented one level. An empty object is just key: with no children.
• Keys matching [A-Za-z_][A-Za-z0-9_.]* are written bare. Any other key is quoted, for example "first name": Ada or "order-id": 1.
• Since 4.0, __proto__, constructor and prototype are ordinary keys. Decoders must not let them pollute object prototypes.

4. Arrays

Every array header declares its length in brackets. The encoder picks one of four forms: inline primitive, tabular, keyed tabular, or expanded list.

4.1 Inline Primitive Arrays

Arrays of primitives go on one line after the header:

tags[3]: featured,new,on-sale

4.2 Expanded List (Heterogeneous)

Used when items are mixed types or objects with different keys. Each item starts with - :

events[2]:
  - type: login
    user: ada
  - type: purchase
    user: bob
    amount: 20

Mixed arrays work the same way. [1, {"a": 1}, "x", [1, 2]] under the key items encodes as:

items[4]:
  - 1
  - a: 1
  - x
  - [2]: 1,2

4.3 Tabular Arrays (Homogeneous)

Used when every item is an object with the same keys and only primitive values. This is TOON's main optimization, and since spec 4.1 encoders MUST use it whenever the data qualifies.

Syntax header: key[N]{field1,field2}:

users[3]{id,name,score}:
  1,Alice,99
  2,Bob,85
  3,Charlie,42

Rules:

• Count: [N] is the number of rows. Strict decoders check it, so a header that says 3 with only 2 rows fails with Expected 3 tabular rows, but got 2.
• Fields: The {a,b} list defines the column order for every row.
• Cells: Rows contain only primitive values, with no space after the delimiter.
• Delimiter: Comma by default. Tab and pipe are official alternatives, declared inside the brackets and used in the field list too.

Pipe delimiter, from encode(data, { delimiter: "|" }):

users[2|]{id|name|score}:
  1|Alice|99
  2|Bob|85

Tab delimiter (delimiter: "\t") uses the same layout with a literal tab character after the count, between the field names, and between the cells: users[2<TAB>]{id<TAB>name<TAB>score}:. Tabs often tokenize well, and values containing commas don't need quotes. For when to pick each one, see comma vs tab vs pipe delimiters.

4.4 Keyed Tabular Form (new in 4.0)

An object whose values are all objects with the same primitive fields (a map such as feature flags) gets a table where each row starts with its key. A colon after the count marks the keyed form:

flags[3:]{enabled,rollout}:
  darkMode: true,50
  newCheckout: false,0
  betaSearch: true,10

At the root the header has no key: [2:]{enabled,rollout}:. This decodes back to {"flags": {"darkMode": {"enabled": true, "rollout": 50}, ...}}.

4.5 Nested Field Groups (new in 4.0)

If every row has a nested object with the same primitive fields, the header lists its fields in braces after the field name. The array stays tabular, and each row carries the nested values inline:

orders[2]{id,customer{name,country},total}:
  1,Ada,UK,49.99
  2,Bob,US,199

Note that 199.0 in the source JSON is written as 199 (see 5.2).

4.6 Empty Arrays

Since spec 3.1, an empty array is written key: [] (or [] at the root). The legacy key[0]: form MUST NOT be emitted, although decoders still accept it.

tags: []

5. Values and Types

5.1 Strings

Strings can be Unquoted or Quoted.

Unquoted Strings (Bare Words): Allowed when the value is not empty, has no leading or trailing whitespace, does not look like a boolean, number or null, does not start with - or #, and contains no colon, quote, backslash, bracket, brace, control character, or the active delimiter.

status: active
color: light blue

Quoted Strings: Double quotes are required otherwise. Escapes are limited to \\, \", \n, \r and \t.

greeting: "Hello, World!"
empty: ""
number_string: "123"
bool_string: "true"
note: "#not a comment"
padded: " x "

The # rule is new in 4.0. A string that starts with # must be quoted, so it can never be confused with a comment line.

5.2 Numbers

Encoders write numbers in canonical decimal form: no exponent where a plain decimal works, no trailing zeros, and -0 becomes 0. For {"count": 42, "temp": 36.6, "ratio": 1.0, "big": 1e6, "neg": -0}:

count: 42
temp: 36.6
ratio: 1
big: 1000000
neg: 0

Spec 4.0 also defines a normative number grammar for decoders. Forms such as .5, 1., +5, 1_000, 0x10, Infinity and NaN are not numbers, so they decode as strings. NaN and infinities in the input encode as null.

5.3 Booleans

Literals true and false (lowercase).

5.4 Null

Literal null.

6. Comments (new in 4.0)

Comments did not exist before spec 4.0. They are now allowed, but only as full-line comments: a line whose first non-space character is #. In the spec's words, "a '#' anywhere else on a line is ordinary content, and no inline or trailing comment form exists." Encoders MUST NOT emit comments. They are for hand-written TOON only.

# Service config
config:
  # milliseconds
  timeout: 5000

This decodes to {"config": {"timeout": 5000}}. A trailing # does not start a comment: timeout: 5000 # milliseconds decodes as the string "5000 # milliseconds" (verified with @toon-format/toon 4.1.1).

7. ABNF Grammar (Simplified Excerpt)

The following is a simplified, non-normative sketch of the line grammar. It leaves out quoting, escaping and the full number rules. See SPEC.md for the authoritative definition.

document     = *(line LF) line
line         = *indent (comment / pair / list-item / header / row)
indent       = 2SP                      ; indentSize, default 2
comment      = "#" *VCHAR-OR-SP         ; full line only

pair         = key ":" [SP primitive]
list-item    = "- " (primitive / pair / header)
header       = [key] "[" count [":"] [delim] "]" [fields] ":" [SP inline-values]
fields       = "{" field *(delim field) "}"
field        = key ["{" field *(delim field) "}"]   ; nested field group
row          = [key ": "] primitive *(delim primitive)  ; key only in [N:] tables
inline-values = primitive *(delim primitive)

delim        = "," / "|" / HTAB          ; declared in the header
primitive    = string / number / "true" / "false" / "null"

8. Parsing Implementation Guide

When implementing a TOON parser, the primary challenge is Context Tracking. Since structure is defined by indentation, the parser must maintain a stack of current indentation levels and the active array header (count, fields and delimiter).

Algorithm Sketch:

1. Read a line and strip a leading BOM from the first line. Skip blank lines and full-line # comments.
2. Calculate the indentation level (leading spaces divided by the indent size). In strict mode, reject tabs in indentation and depth jumps of more than one level.
3. If the level is lower than the current level, pop containers until it matches.
4. Classify the content: array header, list item, tabular row (or keyed row), or key-value pair.
5. When an array closes, check that the number of items or rows matches the declared [N].

The reference decoder throws ToonDecodeError (since library 2.2.0) with a line number, for example Line 2: Tabs are not allowed in indentation in strict mode.

9. Test Suite

To ensure interoperability, implementations should pass the language-agnostic conformance fixtures published in the toon-format/spec repository. They cover:

• Encoding and decoding of every array form, including keyed tables and nested field groups.
• Quoting, escaping, canonical numbers, and Unicode handling (emoji, CJK characters).
• Strict-mode errors: count mismatches, depth jumps, tabs in indentation, and invalid escapes.

For a quick-reference version of these rules, see the TOON syntax cheat sheet. For how nesting is handled in practice, see how TOON handles nested data.