Running TOON with Ollama and Local LLMs

Local models on Ollama — Llama, Phi, Qwen — typically ship with context windows of 2k to 8k tokens. TOON cuts token usage by up to 58.8% on flat uniform tables and 39.9% overall versus JSON, letting a constrained local context hold roughly twice as many rows of data without any model change or hardware upgrade.

Why Context Window Size Is the Bottleneck for Local Models

Cloud APIs like Gemini and GPT-5 offer context windows measured in hundreds of thousands of tokens. Most local models running on consumer hardware through Ollama are quantized to 4-bit or 8-bit precision and loaded with default context lengths of 2,048 to 8,192 tokens. Even models that support longer contexts (Llama 3.1 supports up to 128k in theory) are often capped lower in practice to keep VRAM usage manageable.

When you feed tabular data into one of these models as JSON, a significant fraction of those tokens are structural overhead: repeated keys, braces, quotes, and commas that appear on every row but carry no unique information. On a uniform array of 50 records with five fields each, JSON might spend 60–70% of its token budget on punctuation and key repetition. TOON eliminates that by declaring field names once in the header and emitting only values per row.

According to the official TOON benchmarks (5,016 LLM calls across 209 questions and four models), TOON uses 39.9% fewer tokens overall versus JSON and up to 58.8% fewer on flat uniform tables. That compression is format-level — it applies regardless of which model reads the prompt. The context-window math is straightforward: what previously filled a 4k window now fits in roughly 1.7k tokens, leaving room for the system prompt, the question, and the model's response.

How Many Rows Fit in a Local Context Window: JSON vs TOON

The table below estimates capacity for a representative five-field row schema (id, name, status, value, timestamp) at typical Ollama default context lengths. Token estimates assume the GPT-5 o200k_base tokenizer; actual counts vary by tokenizer and content.

* Usable tokens after reserving ~350 tokens for system prompt, question, and response buffer. TOON estimate includes ~80 tokens for the format header and brief instruction. Token-per-row figures are approximations; use the converter to measure your actual data.

At every context size, TOON fits roughly twice as many rows as JSON. For a model like Phi-3 Mini with a 2k default context, this is the difference between fitting 45 rows of a log table versus 100 — which can determine whether the model has enough evidence to answer a question correctly.

Ollama Chat API: JSON Payload vs TOON Payload

The Ollama /api/chat endpoint works identically to the OpenAI chat format. You place TOON-encoded data in the message content the same way you would JSON — the only difference is the format of the data string itself and a brief system-prompt instruction.

# ── JSON version ── (curl, ~310 tokens total for 8 log rows)
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [
    {
      "role": "system",
      "content": "You are a data analyst. Answer questions about the provided data."
    },
    {
      "role": "user",
      "content": "Here are recent error logs in JSON:\n[\n  {\"id\":1,\"service\":\"auth\",\"level\":\"error\",\"code\":500,\"ts\":\"2026-06-01T08:12:00Z\"},\n  {\"id\":2,\"service\":\"api\",\"level\":\"warn\",\"code\":429,\"ts\":\"2026-06-01T08:13:10Z\"},\n  {\"id\":3,\"service\":\"auth\",\"level\":\"error\",\"code\":500,\"ts\":\"2026-06-01T08:15:42Z\"},\n  {\"id\":4,\"service\":\"db\",\"level\":\"error\",\"code\":503,\"ts\":\"2026-06-01T08:17:05Z\"},\n  {\"id\":5,\"service\":\"api\",\"level\":\"error\",\"code\":500,\"ts\":\"2026-06-01T08:18:33Z\"},\n  {\"id\":6,\"service\":\"auth\",\"level\":\"warn\",\"code\":401,\"ts\":\"2026-06-01T08:20:11Z\"},\n  {\"id\":7,\"service\":\"db\",\"level\":\"error\",\"code\":503,\"ts\":\"2026-06-01T08:21:48Z\"},\n  {\"id\":8,\"service\":\"api\",\"level\":\"info\",\"code\":200,\"ts\":\"2026-06-01T08:22:00Z\"}\n]\n\nWhich services have the most error-level events?"
    }
  ],
  "stream": false
}'

# ── TOON version ── (curl, ~130 tokens total — ~58% fewer)
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [
    {
      "role": "system",
      "content": "You are a data analyst. The user will provide data in TOON format: arrays are written as name[count]{field1,field2,...}: followed by one comma-separated value row per line. Answer questions about the provided data."
    },
    {
      "role": "user",
      "content": "Here are recent error logs in TOON format:\n\nlogs[8]{id,service,level,code,ts}:\n  1, auth, error, 500, 2026-06-01T08:12:00Z\n  2, api, warn, 429, 2026-06-01T08:13:10Z\n  3, auth, error, 500, 2026-06-01T08:15:42Z\n  4, db, error, 503, 2026-06-01T08:17:05Z\n  5, api, error, 500, 2026-06-01T08:18:33Z\n  6, auth, warn, 401, 2026-06-01T08:20:11Z\n  7, db, error, 503, 2026-06-01T08:21:48Z\n  8, api, info, 200, 2026-06-01T08:22:00Z\n\nWhich services have the most error-level events?"
    }
  ],
  "stream": false
}'

The TOON version moves the field names out of every row and into a single header line (logs[8]{id,service,level,code,ts}:). The system prompt adds roughly 50 tokens of format explanation, but the data itself is far more compact. At eight rows the savings are modest; at 80 rows they free up the majority of a small local context window.

To convert your own JSON data to TOON before pasting it into a prompt, use the free json2toon.co converter — conversion happens client-side and nothing leaves your browser. For a full description of TOON syntax, see What is TOON?

Accuracy Varies by Model: Test Before You Deploy

The official TOON benchmark tested cloud models — not local Ollama models — but its per-model accuracy spread is an important caution for local deployments. Among the four models tested, TOON retrieval accuracy ranged from 96.7% on Gemini 3 Flash down to 58.4% on Grok 4.1 Fast, according to the toonformat.dev benchmarks. That 38-point gap shows that a model's ability to parse TOON correctly is not guaranteed — it depends heavily on how well the model has internalized structured tabular formats during training.

Local models, particularly smaller quantized variants (3B, 7B parameters), have seen less training data overall and may struggle more with unfamiliar formats. A model like Llama 3.2 3B may score 60% on TOON field retrieval where a larger model scores 90%+. The practical rule: run a sample of 20–30 representative questions against your TOON-formatted data before adopting the format in production. If accuracy is below 85%, either switch to a larger local model, increase the detail of your system-prompt format instructions, or fall back to JSON.

For per-model accuracy comparisons and the full benchmark methodology, see the TOON benchmarks guide.

When TOON Pays Off on Local Models — and When It Does Not

A 2026 arXiv paper (2603.03306) — "Token-Oriented Object Notation vs JSON: A Benchmark of Plain and Constrained Decoding Generation" — introduced the concept of the prompt tax: the token overhead of the format instructions you must include to teach a model TOON. The paper found TOON's efficiency is non-linear: it pays off only beyond a threshold where cumulative per-row savings amortize the upfront instruction overhead. On small payloads, the tax can exceed the savings, making plain JSON more efficient overall.

For local models with small context windows, this threshold matters more, not less. A 50-token instruction overhead on a 2k context budget represents 2.5% of your total budget. That overhead is worth paying only when the data itself is large enough to recoup it — roughly 20+ rows for a five-field schema based on TOON's ~16-tokens-per-row vs JSON's ~38.

The arXiv paper also confirmed that for generation tasks — asking the local model to produce structured output — plain JSON outperforms TOON on one-shot and final accuracy. Local models have been trained on vastly more JSON than TOON, so their internal representation of JSON structure is stronger. Use TOON exclusively for input context: data you feed the model to read and reason over, not data you ask it to produce.

For a full breakdown of when to use each format, see our JSON vs TOON comparison and the TOON best practices guide.

Practical Recommendations for Ollama Deployments

Distilling the above into a short checklist for developers using Ollama locally:

• Use TOON when your data is a uniform array of 20 or more objects sharing the same schema and the task is retrieval or comprehension — not generation.
• Include a brief system-prompt instruction describing TOON syntax. Roughly 50–80 tokens is sufficient: explain the name[count]{fields}: header format and comma-separated rows. This is the prompt tax, and it is amortized quickly on large tables.
• Use JSON for payloads under 20 rows, nested or irregular data, and any prompt where the model must output structured data.
• Prefer larger local models (7B+) for TOON if accuracy matters. Smaller models may parse the format inconsistently. Test with your actual data before committing.
• Measure your real token counts. Use the converter to generate a TOON version of your dataset and compare sizes. The 39.9% overall and 58.8% flat-table figures are benchmark averages — your data shape determines the actual savings.
• Consider cost savings at the API layer too. If you also use cloud APIs for some tasks, TOON's token reduction applies there as well. See the API cost optimization guide for the full picture.

Frequently Asked Questions