JSON vs TOON for Large Language Models

JSON is the English of the Internet. It is spoken by every server, every browser, and every database. It won the API wars because it was simple, human-readable, and natively supported by JavaScript.

But in 2024, a new consumer has arrived: The LLM. And for an LLM, JSON is a second language that it speaks fluently but inefficiently. Today we are exploring why JSON is suboptimal for AI, and comparing it to TOON, the challenger designed specifically for BPE Tokenizers.

The Hidden "Token Tax" of JSON

To understand why JSON is expensive, you have to understand how LLMs read text. They don't read bytes; they read Tokens.

Most modern models (GPT-4, Claude, Llama 3) use a variation of BPE (Byte Pair Encoding).

{ "id": 123, "status": "active" }

id: 123
status: active

30% Savings on a single object. Now imagine an array of 1,000 objects.

The Array Multiplier Effect

The difference explodes when you process lists of items (Standard RAG payload).

JSON Array

[
  { "id": 1, "name": "Alice" },
  { "id": 2, "name": "Bob" },
  ...
]

For every single user, you pay for the tokens: ` "id` and ` "name`. If you have 1,000 users, you pay for those keys 1,000 times.

TOON Table

users[2]{id, name}:
  1, Alice
  2, Bob
  ...

You pay for `id` and `name` exactly once in the header. The rows are just data.

In large arrays, TOON is ~60% cheaper than JSON.

Beyond Cost: Robustness and Recovery

There is another subtle advantage: Error Recovery.

JSON Fragility

JSON is a strict recursive structure. If you miss a single closing brace `}` at the end of a 10MB file, standard parsers (like `JSON.parse`) will crash and discard everything.

LLMs are notorious for cutting off output mid-stream (max token limit). If an LLM gives you half a JSON object, it is useless.

TOON Resilience

TOON is line-oriented.

users[3]{id, name}:
  1, Alice
  2, Bob
  3, Char... <EOF>

If the stream cuts off after "Char", a robust TOON parser can still recover the first two rows (Alice and Bob). This makes it significantly better for streaming responses.

The Hybrid Integration Pattern

Should you delete all your Postgres/Mongo databases and replace them with TOON? No.

JSON and SQL are optimized for storage and querying. TOON is optimized for Context.

The Golden Path:

1. Storage: Store data in Postgres/Mongo (JSON/BSON).
2. Backend: Fetch data as Objects (JSON).
3. Edge: Just before calling OpenAI/Anthropic, convert the Object to TOON string.
4. Prompt: Inject TOON into prompt.

This gives you the best of both worlds: the mature tooling of the JSON ecosystem and the cost savings of the TOON format.

Conclusion

JSON is not going anywhere. It is the bedrock of the web.
But TOON is the future of the Context Window.

If you are building Agentic Workflows, RAG pipelines, or Data Extraction bots, switching your "Last Mile" serialization to TOON is the single highest-ROI optimization you can make today.