Compressing REST API Responses for LLMs with TOON

To compress a REST API response for an LLM, drop the fields the model does not need, then re-encode the remaining array as TOON before inserting it into the prompt. REST JSON repeats every key, brace, and quote on every record — TOON moves the field names to a single header line and writes bare values per row, saving up to 58.8% of tokens on uniform arrays while keeping field retrieval accuracy at 99.6%.

Why REST JSON Is Wasteful in LLM Prompts

REST endpoints return fixed structures. A /orders endpoint returns the same shape regardless of what the caller actually needs: order ID, customer name, line items, shipping address, payment method, timestamps, and a dozen internal fields the LLM will never reference. This over-fetching inflates the response before you even consider serialization.

But even after you filter to only the fields the model needs, standard JSON continues to waste tokens. Every object in an array repeats every key name, every pair of double quotes, every colon, every brace. A token is roughly 0.75 English words, and structural characters each cost at least one token. On an array of 200 order objects with five fields, the repeated syntax can account for well over half the total token count — tokens that carry zero information for the model.

TOON addresses this at the serialization layer. The format was designed by Johann Schopplich and contributors and released MIT-licensed in late 2025. Its core idea: declare field names once in a header line (orders[200]{id,customer,total,status,date}:), then write each record as a bare comma-separated row. The per-row structural overhead collapses to a single delimiter character.

For a foundational explanation of what TOON is and how the header syntax works, see our TOON format overview.

How Much Does TOON Save on REST Array Responses?

The official toonformat.dev benchmarks ran 5,016 LLM calls across 209 questions, six formats, and four models (GPT-5 Nano, Claude Haiku, Gemini 3 Flash, Grok 4.1 Fast), using the GPT-5 o200k_base tokenizer. Key results:

• Overall: TOON used 39.9% fewer tokens than JSON while raising retrieval accuracy from 75.0% to 76.4%.
• Flat uniform arrays: 58.8% fewer tokens (67,778 vs 164,452 tokens) — the shape most REST list endpoints return.
• E-commerce orders (nested): 33.3% fewer tokens (73,126 vs 109,599 tokens).
• Efficiency ratio: TOON delivers 27.7 accuracy-points per 1,000 tokens versus JSON's 16.4.
• Field retrieval accuracy: 99.6% — essentially perfect for lookup tasks.

The flat-array figure is particularly relevant to REST. Most paginated list endpoints — /users, /products, /transactions — return arrays of objects sharing an identical schema with no deep nesting. That is precisely the data shape where TOON's per-row savings are largest.

REST JSON vs Field-Trimmed JSON vs TOON: A Comparison

For a full head-to-head token count across formats, see our JSON vs TOON comparison.

A Concrete Example: REST Response Before and After TOON

Consider a /api/orders endpoint that returns recent orders. After fetching, filter to the fields the LLM needs and encode the result as TOON.

The raw REST JSON response (field-trimmed to the five relevant columns):

// Field-trimmed REST JSON — keys repeat on every object (~120 tokens for 5 records)
[
  { "id": "ord-001", "customer": "Alice Chen",  "total": 142.50, "status": "shipped",    "date": "2026-05-28" },
  { "id": "ord-002", "customer": "Bob Nguyen",  "total": 89.00,  "status": "processing", "date": "2026-05-29" },
  { "id": "ord-003", "customer": "Carol Diaz",  "total": 310.75, "status": "delivered",  "date": "2026-05-27" },
  { "id": "ord-004", "customer": "David Park",  "total": 55.20,  "status": "shipped",    "date": "2026-05-30" },
  { "id": "ord-005", "customer": "Emma Wilson", "total": 198.00, "status": "processing", "date": "2026-05-30" }
]

The same data encoded as TOON (field names declared once in the header):

# TOON encoding — ~55 tokens for the same 5 records (~54% reduction)
orders[5]{id,customer,total,status,date}:
  ord-001, Alice Chen,  142.50, shipped,    2026-05-28
  ord-002, Bob Nguyen,  89.00,  processing, 2026-05-29
  ord-003, Carol Diaz,  310.75, delivered,  2026-05-27
  ord-004, David Park,  55.20,  shipped,    2026-05-30
  ord-005, Emma Wilson, 198.00, processing, 2026-05-30

On five records the saving is already over 50%. On a paginated response of 100 records the ratio approaches the 58.8% benchmark figure as the single-line header overhead becomes negligible. Paste any REST response into the free json2toon.co converter to see the exact token count for your data.

How to Encode a REST Response as TOON in Code

The encoding step is a one-line call after the fetch resolves. Install the @toon-format/toon package and call encode on the array:

import { encode } from "@toon-format/toon";

// Fetch and field-trim the REST response
const response = await fetch("https://api.example.com/orders?limit=100");
const json = await response.json();

// Keep only the fields the LLM needs
const trimmed = json.map(({ id, customer, total, status, date }) => ({
  id, customer, total, status, date,
}));

// Encode as TOON for the prompt
const toonOrders = encode(trimmed);

const prompt = `
You are an order-management assistant.
Recent orders:
${toonOrders}

Identify any orders that have been in "processing" for more than 24 hours.
`;

The field-trimming step (the map call) handles the over-fetching problem that REST's fixed structures create. The encode call handles the serialization problem. Both steps are pure JavaScript, run in Node.js or the browser, and add negligible latency. For a full migration walkthrough, see our guide on migrating from JSON to TOON.

Why Tabular Encoding Also Improves LLM Accuracy

Token savings are the primary motivation, but there is a second benefit. A 2026 arXiv study (arXiv 2412.17189) — "Talking with Tables for Better LLM Factual Data Interactions" — found that presenting data as tabular structures produces a 40.29% average performance gain over semi-structured formats (JSON, knowledge graphs) and text-blended formats. Attention analysis showed that tables guide models to attend to relevant information more directly.

TOON's table block (array[n]{fields}: header plus value rows) is structurally a table. The header provides an explicit in-context schema — field names, array length — that the model can verify against the data rows. This is why the TOON benchmark reports 99.6% field retrieval accuracy even at large array sizes. JSON's repeated keys technically carry the same information, but the tabular layout processes better.

For production pipelines where you are feeding retrieved documents or database results into an LLM, the TOON RAG pipeline guide covers the same pattern applied to vector-store results. The API cost optimization guide quantifies the dollar savings from combining TOON with prompt caching and batch processing.

When Not to Encode REST Responses as TOON

TOON's efficiency gains are non-linear. A February 2026 arXiv paper (2603.03306) identifies a prompt tax: the token cost of the format instructions the model needs to interpret TOON correctly. For small payloads this overhead can exceed the per-row savings. The paper formalizes this as a threshold effect — TOON's efficiency pays off only once the cumulative per-row savings amortize the upfront instruction cost.

Use plain JSON (field-trimmed) instead of TOON when:

• The array has fewer than 10 objects. The prompt-tax overhead is not amortized over enough rows.
• The response is highly nested or non-uniform. TOON saves only 21.9% on mixed structures versus 58.8% on flat arrays. Evaluate whether the saving justifies the format overhead.
• The LLM must produce structured output. For generation tasks, JSON remains superior. The arXiv study found plain JSON had the best one-shot and final generation accuracy. Use TOON for the input context, JSON for the output schema.
• The REST endpoint already supports field selection (sparse fieldsets, OData $select). Trim at the API layer first, then decide whether TOON adds enough additional saving.

The sibling post on feeding GraphQL responses to LLMs with TOON covers the pattern for field-scoped APIs, where GraphQL's selection set handles the over-fetching layer before TOON handles the serialization layer.

Frequently Asked Questions