TONL Guide: Token-Optimized Language for LLMs

TONL (Token-Optimized Notation Language) is a production-ready data platform that combines compact serialization with powerful query, modification, indexing, and streaming capabilities. Designed specifically for LLM token efficiency, TONL can reduce your token usage by up to 45% compared to standard JSON, while providing a rich API for data access and manipulation.

What is TONL?

TONL is a data serialization format that goes beyond simple conversion. While formats like TOON focus on compact representation, TONL provides a complete data platform with:

• Compact Serialization: 32-45% smaller than JSON in both bytes and tokens
• Query API: JSONPath-like queries with filtering, aggregation, and fuzzy matching
• Modification API: Full CRUD operations with change tracking
• Indexing: Hash, BTree, and compound indexes for fast lookups
• Streaming: Handle multi-GB files with minimal memory
• Schema Validation: TSL (TONL Schema Language) with 13 built-in constraints

TONL Format Overview

TONL uses a tabular format for arrays of objects, which is perfect for structured data. Here's a comparison:

JSON Example (245 bytes, 89 tokens):

{
  "users": [
    { "id": 1, "name": "Alice", "role": "admin" },
    { "id": 2, "name": "Bob, Jr.", "role": "user" },
    { "id": 3, "name": "Carol", "role": "editor" }
  ]
}

TONL Example (158 bytes, 49 tokens - 45% reduction):

#version 1.0
users[3]{id:u32,name:str,role:str}:
  1, Alice, admin
  2, "Bob, Jr.", user
  3, Carol, editor

As you can see, TONL eliminates redundant syntax while maintaining full type information and human readability.

Key Features of TONL

1. Advanced Query Capabilities

TONL includes a powerful query API that supports JSONPath-like syntax:

// Query examples
doc.get('users[0].name');                          // 'Alice'
doc.query('users[*].name');                        // ['Alice', 'Bob', 'Carol']
doc.query('users[?(@.role == "admin")]');          // Filter by role
doc.query('$..age');                               // Recursive search

// Aggregation
doc.count('users[*]');                             // 3
doc.sum('users[*]', 'age');                        // Sum of ages
doc.avg('users[*]', 'age');                        // Average age
doc.groupBy('users[*]', 'role');                   // Group by role

2. Fuzzy Matching & Temporal Queries

TONL includes advanced query features like fuzzy string matching and temporal queries:

import { fuzzySearch, soundsLike } from 'tonl/query';
fuzzySearch('Jon', ['John', 'Jane', 'Bob']);       // Fuzzy match
soundsLike('Smith', 'Smyth');                      // Phonetic match

import { parseTemporalLiteral, isDaysAgo } from 'tonl/query';
parseTemporalLiteral('@now-7d');                   // 7 days ago
isDaysAgo(someDate, 30);                           // Check if within 30 days

3. Advanced Optimization

TONL includes multiple optimization strategies that can achieve up to 60% additional compression:

• Dictionary Encoding: String deduplication with reference encoding (30-50% savings)
• Delta Encoding: Sequential value compression (40-60% savings)
• Bit Packing: Boolean and small integer bit-level compression (87.5% savings)
• Run-Length Encoding: Repetitive value compression (50-80% savings)
• Adaptive Optimizer: Automatically selects the best strategy

4. Schema Validation

TONL includes a powerful schema language (TSL) with 13 built-in constraints:

@schema v1
@strict true

User: obj
  id: u32 required
  username: str required min:3 max:20 pattern:^[a-zA-Z0-9_]+$
  email: str required pattern:email lowercase:true
  age: u32? min:13 max:150
  roles: list<str> required min:1 unique:true

users: list<User> required min:1

TONL vs TOON: When to Use Each

Both TONL and TOON are designed for token efficiency, but they serve different use cases. For a detailed comparison, see our complete TOON vs TONL guide.

• Use TOON when you need a simple, lightweight format for basic data conversion. It's perfect for straightforward JSON-to-optimized conversions.
• Use TONL when you need advanced features like querying, indexing, schema validation, or when working with large datasets that require streaming or optimization.

Performance Comparison

Based on typical e-commerce product catalog data:

Getting Started with TONL

You can start using TONL right now with our online converter. Simply paste your JSON data and select TONL as the target format. The converter will automatically optimize your data for maximum token efficiency. For programmatic use, check out our TypeScript/JavaScript installation guide.

For programmatic use, install the TONL package:

npm install tonl

import { encodeTONL, decodeTONL } from 'tonl';

const data = { users: [{ id: 1, name: "Alice" }] };
const tonlText = encodeTONL(data);
const restored = decodeTONL(tonlText);

Conclusion

TONL represents the next evolution in token-optimized data formats. With its combination of compact serialization, powerful query capabilities, and advanced optimization features, TONL is the ideal choice for developers building LLM-powered applications that need both efficiency and functionality.

Whether you're building AI agents, optimizing API costs, or processing large datasets, TONL provides the tools you need to maximize efficiency while maintaining code clarity and data integrity.

Try converting your data to TONL format today and see how much you can save on your LLM API costs! Learn more about optimizing API costs or compare TONL with TOON.