Feeding GraphQL Responses to LLMs with TOON

The best way to send GraphQL data to an LLM is to apply two compression passes: first, a precise GraphQL selection set that fetches only the fields the model needs; second, TOON encoding that collapses the repeated keys and punctuation JSON pays on every record. Each pass is independent — together they cut tokens twice and keep field retrieval accuracy at 99.6%.

Why GraphQL Alone Is Not Enough for LLM Prompts

GraphQL solves the field-selection problem. REST endpoints return fixed structures, so a client asking for a product name gets the name plus the price, inventory count, supplier ID, and 14 other fields it never asked for. Enterprise benchmarks show that switching REST to GraphQL reduced payload sizes by 40–60% on average for mobile clients (30–50% more conservatively) simply by eliminating this over-fetching. That is a meaningful reduction before you touch serialization.

The problem is that a well-scoped GraphQL response is still JSON. If that response contains an array of 200 product objects, each object repeats every key name — "id", "name", "price" — wrapped in braces and quotes 200 times. A token is roughly 0.75 English words, and every structural glyph costs one. On a uniform array of objects, JSON's per-row punctuation tax compounds across every record.

TOON solves the serialization problem. It declares field names once in a header (products[200]{id,name,price}:) and writes bare values row by row. The two techniques address different layers of redundancy, so they stack without overlap.

How Much Does Each Layer Save?

The token-reduction figures for TOON come from the official toonformat.dev benchmarks: 5,016 LLM calls across 209 questions, six formats, and four models (GPT-5 Nano, Claude Haiku, Gemini 3 Flash, Grok 4.1). On flat uniform arrays TOON used 67,778 tokens against JSON's 164,452 — a 58.8% reduction. Overall across all data shapes it saved 39.9% of tokens while raising retrieval accuracy from 75.0% to 76.4% and delivering 27.7 accuracy-points per 1,000 tokens versus JSON's 16.4.

A Concrete Example: GraphQL Query, JSON Response, and TOON Encoding

Suppose you are building a product-recommendation feature and need to feed a catalog to an LLM. The GraphQL query requests only the fields the model needs:

# GraphQL selection set — only four fields, no over-fetching
query GetProducts {
  products(first: 5, category: "electronics") {
    id
    name
    price
    inStock
  }
}

The server returns a standard JSON response:

// GraphQL JSON response — keys repeat on every object
{
  "data": {
    "products": [
      { "id": "p01", "name": "Wireless Headphones", "price": 79.99, "inStock": true },
      { "id": "p02", "name": "USB-C Hub",           "price": 34.50, "inStock": true },
      { "id": "p03", "name": "Mechanical Keyboard",  "price": 129.00, "inStock": false },
      { "id": "p04", "name": "Webcam 1080p",         "price": 59.95, "inStock": true },
      { "id": "p05", "name": "Monitor Stand",         "price": 45.00, "inStock": false }
    ]
  }
}

Before inserting into the LLM prompt, encode the array as TOON. The field names move to the header; each row becomes a bare comma-separated line:

# TOON encoding — keys declared once, rows are pure values
products[5]{id,name,price,inStock}:
  p01, Wireless Headphones, 79.99, true
  p02, USB-C Hub, 34.50, true
  p03, Mechanical Keyboard, 129.00, false
  p04, Webcam 1080p, 59.95, true
  p05, Monitor Stand, 45.00, false

The JSON version uses approximately 105 tokens for this five-item response. The TOON version uses approximately 55 tokens — a 47% reduction on a small array. On a 200-item response the gap approaches the 58.8% benchmark figure as the per-header overhead amortizes fully across the rows.

Use the free json2toon.co converter to encode any GraphQL JSON response in your browser — no data leaves your machine.

Where in the Pipeline to Apply TOON

GraphQL clients (Apollo Client, urql, Relay, or a plain fetch) always return JSON. TOON is a post-processing step: after the client resolves the promise, extract the relevant array from data and serialize it with the @toon-format/toon package before building the prompt string.

import { encode } from "@toon-format/toon";

// After your GraphQL client resolves:
const { data } = await graphqlClient.query({ query: GET_PRODUCTS });

// Extract the array, encode as TOON, insert into the prompt
const toonProducts = encode(data.products);

const prompt = `
You are a product recommendation assistant.
Available products:
${toonProducts}

Recommend the best option for a home office setup.
`;

The pattern works identically with server-side GraphQL (a Node.js route that fetches from a remote API) and with client-side GraphQL (a React component using a hook). The encoding step is pure CPU, no I/O, and typically takes under 1ms for arrays under 10,000 rows.

For deeper advice on building cost-efficient LLM calls, see our guide on optimizing API costs with TOON and the TOON RAG pipeline guide, which covers similar encoding patterns for retrieved context chunks.

Caveats: When to Skip TOON on GraphQL Responses

TOON's efficiency is non-linear. A February 2026 arXiv study (2603.03306) on TOON vs JSON benchmarks confirms a prompt tax: the instructional overhead required to teach the model the format. For small responses the overhead can exceed the savings.

Practical thresholds for GraphQL responses:

• Fewer than 10 objects: use the raw GraphQL JSON. The prompt tax erases the per-row savings on short lists.
• Highly nested responses (e.g., orders with line items with variants): TOON's table syntax helps less on non-repetitive nesting. On mixed structures the benchmark shows only a 21.9% reduction rather than 58.8%.
• LLM generation tasks: if you need the model to produce structured output, use JSON. The arXiv study found JSON has the best one-shot and final accuracy for generation. Use TOON only for the input context.
• Deeply nested GraphQL unions or interfaces: flatten to a uniform shape first, then encode as TOON. Non-uniform types in a single array undercut the header's amortization.

For a full breakdown of data-shape trade-offs, see our JSON vs TOON comparison and the companion post on compressing REST API responses for LLMs.

Why TOON Tables Improve LLM Accuracy on Structured Data

The token savings are the most visible benefit, but there is a second mechanism worth understanding. A 2026 arXiv paper (arXiv 2412.17189) — "Talking with Tables for Better LLM Factual Data Interactions" — found that presenting data as tabular structures yields a 40.29% average performance gain over text-blended and semi-structured formats. Attention analysis showed that tables help models attend to relevant information more directly.

TOON's core encoding is exactly a tabular block: a header line declaring field names and row count, followed by value-only rows. This aligns with how LLMs process structured data most effectively. GraphQL gives you precise field selection; TOON gives you the tabular layout that maximizes what the model does with those fields.

This is also why the TOON benchmark reports 99.6% field retrieval accuracy — the explicit field header doubles as an in-context schema that the model can verify against each row. The TOON format overview explains this design in detail.

Frequently Asked Questions