LLM Log Analysis at Scale: Why TOON Beats Raw JSON Logs

TOON is the best format for feeding structured logs to an LLM. Application logs are uniform by design — every line shares the same schema — which is exactly where TOON's header-once table block eliminates the most waste. Official benchmarks record 59.0% fewer tokens on time-series data versus JSON, with 99.6% field retrieval accuracy. For log volumes too large for a single context window, TONL adds streaming and a query API on top.

Why JSON Is a Poor Fit for Log Data

Structured logs are almost always an array of objects sharing the same schema: timestamp, level, service, message, maybe a request ID or latency figure. When you serialize that as JSON, every single line pays the full per-key tax — the field names in curly braces, surrounded by double quotes, separated by colons and commas — repeated identically thousands of times.

LLMs use Byte Pair Encoding tokenizers. Every repeated structural glyph — {, }, ", :, , — costs tokens on every object. On an array of N log entries you pay for the keys and punctuation N times. The official toonformat.dev benchmarks quantify this directly: on flat uniform tables, JSON used 164,452 tokens versus TOON's 67,778 — a 58.8% reduction. On 60-day time-series data, the gap is nearly as large at 59.0% (22,245 vs 9,115 tokens).

NDJSON — writing one JSON object per line — is a common improvement for log pipelines because it enables line-by-line streaming without loading the full array. But NDJSON still repeats every field name on every line. It solves a parsing problem, not a token problem.

NDJSON vs TOON: Three Log Entries Side by Side

The difference is easiest to see with a concrete example. Here are three structured log entries in NDJSON followed by the equivalent TOON table block.

// NDJSON — field names repeated on every line
{"ts":"2026-06-01T08:00:01Z","level":"INFO","service":"api-gateway","status":200,"latency_ms":34,"msg":"GET /health"}
{"ts":"2026-06-01T08:00:04Z","level":"WARN","service":"auth-service","status":401,"latency_ms":12,"msg":"Invalid token"}
{"ts":"2026-06-01T08:00:09Z","level":"ERROR","service":"db-proxy","status":500,"latency_ms":980,"msg":"Connection timeout"}

// TOON — field names declared once; rows are pure values
logs[3]{ts,level,service,status,latency_ms,msg}:
  2026-06-01T08:00:01Z, INFO,  api-gateway,  200, 34,  GET /health
  2026-06-01T08:00:04Z, WARN,  auth-service, 401, 12,  Invalid token
  2026-06-01T08:00:09Z, ERROR, db-proxy,     500, 980, Connection timeout

The NDJSON version above is 279 characters; the TOON version is 215 characters — and that gap widens dramatically as the row count grows, because the header is declared only once regardless of how many rows follow. At a thousand log entries the NDJSON overhead is three orders of magnitude larger than the one-time TOON header cost.

The tabular representation also helps the model reason about the data. An independent study published on arXiv (arXiv 2412.17189) found that providing data in tabular structures yields a 40.29% average performance gain over semi-structured and text-blended formats — and that LLMs attending to tabular data are better at extracting relevant fields. TOON's core structure is exactly a tabular block.

How Much Context Window Do You Actually Save?

The official benchmarks ran 5,016 LLM calls across 209 questions, six formats, and four models using the GPT-5 o200k_base tokenizer. Across all data shapes, TOON achieved 39.9% fewer tokens overall while matching or beating JSON on retrieval accuracy: 76.4% vs 75.0%. That translates to 27.7 accuracy-points per thousand tokens for TOON against JSON's 16.4.

For logs specifically, the relevant shapes are flat uniform tables (58.8% reduction) and time-series (59.0% reduction). Concretely: if a 60-day server log in JSON costs 22,245 tokens, the same data in TOON costs 9,115 tokens — saving over 13,000 tokens per query. At GPT-5 pricing, that saving compounds on every single LLM call you make against those logs. See the guide to optimizing API costs for a full cost breakdown.

Field retrieval accuracy with TOON sits at 99.6% — essentially perfect for the most common log analysis task: "find me all ERROR entries from service X between time A and B." Structure awareness is 89.0%. The weaker points are aggregation (61.9%) and filtering (56.8%), which matters for log analytics. If your LLM task is pure triage and retrieval, those numbers are excellent. If you need the model to perform complex aggregations, consider pre-filtering with a query layer first.

Comparison Table: Raw JSON vs NDJSON vs TOON vs TONL for LLM Log Triage

For a deeper look at how TOON and TONL differ in capability, see TOON vs TONL: which format should you use?

Handling Large Log Volumes: When to Reach for TONL

TOON's efficiency is non-linear. It pays off when cumulative per-row savings amortize the upfront format-instruction overhead — and for logs with hundreds or thousands of entries sharing the same schema, that threshold is easily crossed. But TOON is still a text format you load into a context window. If you have days or weeks of logs, you cannot fit them all.

That is where TONL steps in. According to tonl.dev, TONL streams files larger than 50 GB in under 100 MB of memory and provides a SQL-like query API with indexed lookups under 0.1 milliseconds. The practical workflow for large-scale log analysis becomes:

1. Store logs in TONL format.
2. Use the TONL query API to filter to the time window and severity level you care about.
3. Serialize that projection to TOON (or keep it in TONL) and send it to the LLM.
4. Ask the model to triage, summarize, or identify anomalies in a token-efficient slice of your logs.

This approach means the LLM never sees noise — only the rows that are relevant to the question — and those rows are encoded in the most token-efficient format available. The query step is essentially free at sub-millisecond latency.

TONL also ships with 2,300+ tests passing and zero runtime dependencies, making it suitable for production pipelines. For an in-depth look at TONL's architecture, see the TONL architecture overview.

A Note on Model Choice

TOON accuracy varies significantly by model. On the official benchmarks, Gemini 3 Flash reached 96.7% overall accuracy with TOON, GPT-5 Nano hit 90.9%, while Claude Haiku scored 59.8% and Grok 4.1 scored 58.4%. If you are routing log analysis tasks to a smaller or less capable model, run your own field-retrieval test before committing TOON to production. The toonformat.dev benchmarks page has the full per-model breakdown.

For log triage the most important accuracy number is field retrieval, which sits at 99.6% — essentially perfect across all tested models. That means "find me all ERROR events from the auth-service" is a task TOON handles reliably regardless of which model you use.

Frequently Asked Questions