Inside TONL Compression: Dictionary, Delta, RLE & Bit Packing

TONL's base format already saves 32–50% of tokens versus JSON by declaring column headers once instead of repeating them on every row. Seven optional compression layers — Dictionary, Delta, RLE, Bit Packing, Column Reorder, Quantizer, and Schema Inheritance — push that ceiling to roughly 60%, each targeting a specific data shape.

Why a Layered Compression Architecture?

Most serialization formats treat compression as an all-or-nothing toggle: either you gzip the whole payload or you do not. TONL takes a different approach. Because TONL is a columnar text format — each column's values are grouped together in a block — it can apply a different strategy to each column based on that column's statistical profile. A column of user-facing status strings and a column of Unix timestamps need completely different treatments to compress well.

The layers are also composable. Dictionary encoding can reduce a high-cardinality string column to integers, and then RLE can further compress a run of identical integer codes in a single pass. The result is that the total savings depend on how well your specific data fits the available layers, not on a fixed algorithm applied uniformly.

Base TONL without any compression layer already achieves this by eliminating repeated field names. Consider how JSON serializes an array of order records — every object repeats every key on every row. TONL declares the schema once:

# JSON — keys repeated on every row (109,599 tokens for e-commerce orders)
[
  {"id": 1001, "status": "shipped", "amount": 49.99, "region": "us-east"},
  {"id": 1002, "status": "pending", "amount": 12.00, "region": "us-west"},
  {"id": 1003, "status": "shipped", "amount": 99.50, "region": "us-east"}
]

# Base TONL — header declared once (~33% smaller than JSON for nested e-commerce data)
orders[3]{id:u32, status:str, amount:f32, region:str}:
  1001, shipped, 49.99, us-east
  1002, pending, 12.00, us-west
  1003, 99.50, us-east
  | shipped

Adding compression layers shrinks the values further. The sections below explain each layer and the data shape it is designed for.

The Seven TONL Compression Layers Explained

Dictionary Encoding — Repeated Strings

Dictionary encoding is the right layer when a string column has low cardinality: the same values appear many times across many rows. A status column with values like active, inactive, and pending — across hundreds of thousands of rows — is a classic case. Dictionary encoding builds a lookup table at the top of the block, assigns a short integer code to each unique value, and replaces every occurrence with the code. The longer the string and the higher the repetition rate, the greater the savings.

Delta Encoding — Sequential or Monotonic Numbers

Delta encoding stores the difference between consecutive values instead of the absolute value. It is ideal for timestamps, incrementing IDs, or any column where values tend to rise steadily. An event-log table with Unix timestamps at one-second intervals produces a delta column of all 1s — a perfect candidate for a second RLE pass. Even irregular sequences benefit when deltas are small compared to the raw values.

Run-Length Encoding (RLE) — Long Repeated Runs

RLE replaces a run of identical consecutive values with a count-plus-value pair. A sensor table where a device is offline for 10,000 consecutive readings collapses from 10,000 tokens to a single entry. RLE is often the best follow-up layer after Dictionary encoding (integer codes may form long runs) or Delta encoding (many identical deltas compress further).

Bit Packing — Small-Range Integers

Integers that only ever fall within a small range do not need to be stored at full width. A boolean-ish column containing only 0 and 1, or a priority column containing values 1–5, requires far fewer bits per value than a general-purpose integer. Bit packing determines the minimum bit width needed for the observed range and serializes accordingly. Because TONL is a text format, the packed representation is still human-readable in context — the packing is applied at the block level, not as binary encoding.

Column Reorder — Grouping Similar Data

Column reorder does not change individual values; it changes which columns sit next to each other in the serialized output. By placing columns with similar value distributions adjacent, it improves the efficiency of subsequent layers like RLE and Delta that operate on contiguous sequences. It also improves LLM attention on similar attributes by grouping them spatially — a small but measurable readability benefit.

Quantizer — Floating-Point Precision

Floating-point columns often carry more precision than the application actually needs. A price column stored as 49.990000000001 due to floating-point representation errors communicates the same information as 49.99. The Quantizer layer rounds float values to a configurable number of significant digits before serialization. This reduces token count on the value itself and dramatically improves subsequent compression on the column because more values become identical after rounding.

Schema Inheritance — Repeated Nested Shapes

Schema Inheritance targets datasets where the same nested object structure appears repeatedly across multiple blocks or sections. Instead of re-declaring the full schema each time, a block can declare that it inherits from a named parent schema and only list the fields that differ. This is most valuable in hierarchical datasets — for example, a multi-tenant API response where every tenant's data has the same top-level structure with minor per-tenant overrides.

Which Compression Layer Fits Which Data Shape?

The table below maps each layer to its best-fit data shape and the mechanism behind it. Layers can be combined — the order listed reflects the typical application sequence.

For a broader look at how TONL's design compares to TOON, see the TOON vs TONL comparison. For the full architectural picture, the architecture of TONL post covers the columnar block model in detail.

What Does a Compressed TONL Block Look Like?

The example below shows a simplified event-log dataset first in plain JSON, then in base TONL, then with Dictionary and Delta layers active. The savings accumulate at each step.

# Plain JSON — all keys repeated, full timestamps, verbose status strings
[
  {"ts": 1717200000, "event": "login",  "user_id": 101, "region": "us-east"},
  {"ts": 1717200001, "event": "login",  "user_id": 102, "region": "us-east"},
  {"ts": 1717200060, "event": "logout", "user_id": 101, "region": "us-east"},
  {"ts": 1717200120, "event": "login",  "user_id": 103, "region": "us-west"}
]

# Base TONL — header declared once; already 32–50% smaller than JSON
events[4]{ts:u32, event:str, user_id:u32, region:str}:
  1717200000, login,  101, us-east
  1717200001, login,  102, us-east
  1717200060, logout, 101, us-east
  1717200120, login,  103, us-west

# TONL + Dictionary (event: login=0, logout=1; region: us-east=0, us-west=1)
# + Delta on ts column (differences: 0, 1, 59, 60 from base 1717200000)
@dict event: {login:0, logout:1}
@dict region: {us-east:0, us-west:1}
@delta ts base=1717200000
events[4]{ts:u32, event:str, user_id:u32, region:str}:
  0,  0, 101, 0
  1,  0, 102, 0
  60, 1, 101, 0
  120,0, 103, 1

The compressed block is shorter in both character count and token count. The event column went from repeating 5- and 6-character strings to single digits. The timestamp column went from 10-digit Unix values to small deltas. With RLE, the run of three us-east values (now 0, 0, 0) would collapse to a single entry.

This is also why the compression layers are specifically useful for LLM prompts — not just file storage. Fewer tokens means lower cost and more room in the context window for actual instructions and reasoning. As documented on tonl.dev, the full pipeline can push total savings from the base 32–50% range to approximately 60%.

Production Readiness and Dependencies

A reasonable concern when adopting any compression scheme is reliability. The TONL GitHub repository reports 2,300-plus tests passing and zero runtime dependencies for the tonl npm package at v2.5.2. Each compression layer has its own test coverage within that suite.

Zero runtime dependencies matter specifically because compression libraries have a history of pulling in large dependency trees. With TONL, you install one package and the entire pipeline — base format, all seven compression layers, the query API, schema validation, and streaming — is included without further installation.

For context on how TONL compares to TOON (the simpler, no-dependency format for read-only LLM input), see the introduction to TONL and the JSON vs TOON deep-dive.

When Should You Skip the Compression Layers?

The compression layers add a small parsing overhead on the read path. For LLM prompt construction where the data is serialized once and consumed once, the overhead is negligible. But there are cases where base TONL without extra layers is the right choice:

• Small datasets. A block of fewer than 20–30 rows gains little from Dictionary or Delta encoding — the dictionary header costs tokens that may not be recovered from the compressed values.
• High-cardinality string columns. If a column has near-unique values (names, UUIDs, free-text notes), Dictionary encoding produces a lookup table larger than the original data.
• Human-edited files. Compressed TONL is harder to write and review by hand. If your workflow involves people editing the file directly, stick with base TONL or plain JSON.
• One-shot prompts. If you are feeding data into a single prompt rather than building a reusable pipeline, the integration cost of the compression API may not be worth the token savings on that single call.

The right question is always: how many tokens does this column contribute to the prompt, and how often do similar values repeat? If the answer is "a lot" and "often," a compression layer will pay off. Use the free json2toon converter to see base TONL savings before deciding whether to add compression layers.

Frequently Asked Questions