TONL Guide: Token-Optimized Language for LLMs

Data is eating the world. But as we feed more and more data into Large Language Models (LLMs), we are discovering a painful bottleneck: Token Inefficiency.

The standard format for the web, JSON, was designed for JavaScript engines, not for BPE Tokenizers. It is verbose, repetitive, and untyped. Every curly brace, every quote mark, and every repeated key name is a tax on your API bill and your model's attention span.

Today, we are proud to introduce TONL (Token-Optimized Notation Language). TONL is not just a file format; it is a full-stack data platform designed from the ground up to be the Universal Data Plane for the AI Era. Explore the official TONL website for more details.

The Philosophy: Intelligence Requires Efficiency

When you send data to an LLM, you are paying for "Reasoning." But if 60% of your prompt is just JSON syntax (`"id": 1, "id": 2, "id": 3...`), you are forcing the model to spend 60% of its compute just parsing structure.

TONL frees the model to think about the content, not the container.

The Three Pillars of TONL

TONL is built on three core pillars that distinguish it from simple serialization formats like TOON or YAML.

Pillar 1: Extreme Compression

TONL uses advanced techniques borrowed from columnar databases (like Parquet) but applied to text processing.

• Dictionary Encoding: If the string "Pending" appears 1,000 times, TONL writes it once in a shared dictionary and refers to it by a single-byte ID.
• Delta Encoding: For sorted integers (like timestamps or IDs), TONL stores only the difference between values (`100, +1, +2` instead of `100, 101, 103`).
• Bit Packing: Booleans are packed into bits. 8 booleans take 1 byte, not 32 bytes (as `true,false...` would).

Result: Up to 60% token reduction vs JSON.

Pillar 2: Intelligent Typing

JSON is "stringly typed." TONL is Semantically Typed.

We introduce TSL (TONL Schema Language), a human-readable way to define data contracts.

@schema
User {
  id: u64
  email: str pattern:email
  created_at: timestamp
  status: enum(active, inactive, banned)
}

When you feed a TONL file to an LLM, you can optionally include this schema. This gives the model "Ground Truth." It knows exactly what `status` values are valid, reducing hallucinations.

Pillar 3: Random Access & Streaming

Parsing a 1GB JSON file is a nightmare. You have to load the whole thing into RAM.

TONL is designed for Streaming and Random Access.

• Footer Index: A TONL file can have a binary index at the end.
• Range Requests: A client can fetch the last 1KB (the index), find where record #500,000 lives, and then fetch only that byte range.
• Zero-Copy Streaming: The TONL parser (written in Rust/WASM) can filter a multi-gigabyte stream in constant memory, emitting only matching records.

Deep Dive: The Query Engine

TONL isn't just a passive format. It comes with a Query Engine that behaves like a portable database.

// Find high-value users directly from a stream
const results = await tonl.queryStream(fileStream, `
  users[?(@.lifetime_value > 1000 && @.status == "active")]
`);

This allows you to build Agentic Workflows where an Agent can "query" a document without reading the whole thing, saving massive amounts of context window tokens.

The Ecosystem

We are launching with a robust set of tools:

1. The Universal Translator (CLI)

$ tonl convert data.json data.tonl --schema user.tsl
$ tonl query data.tonl "users[*].email"
$ tonl stats data.tonl

2. VS Code Extension

• Syntax Highlighting
• Auto-Completion (based on Schema)
• Live Preview (JSON view)
• Linting errors

3. SDKs

• TypeScript/Node: Production ready.
• Python: Coming next month (with NumPy integration).
• Rust: Core crate available.

Benchmarks: The "Proof is in the Pudding"

We ran a comparison using a dataset of 10,000 e-commerce orders.

TONL hits the sweet spot: usage metrics almost as good as Parquet, but completely text-based and readable by LLMs.

The Vision: Data as a First-Class Citizen

We believe that the future of software is Agents talking to Agents.

Today, they talk in English ("Please give me the user list..."). This is imprecise. Tomorrow, they should talk in TONL.

"Here is the `users.tonl` file. It conforms to `user_schema_v2`. I have indexed it by ID."

This is the future determining the speed and reliability of AI.