Formatting Vector Database Results as TOON for RAG

Format vector database results as a TOON table. Retrieved nodes from Pinecone, Weaviate, or pgvector share a fixed schema — id, score, text, and metadata — which is a uniform array, TOON's best-case shape. The official benchmark shows 58.8% fewer tokens and 99.6% field-retrieval accuracy on uniform arrays, so more evidence fits per context window without sacrificing precision.

Why RAG Context Formatting Directly Affects Answer Quality

A typical RAG pipeline embeds the user query, runs a k-nearest-neighbor search against a vector database with optional metadata filters, and passes the top-k retrieved nodes plus their metadata into the LLM context window to generate a grounded, cited answer. According to the Pinecone RAG guide, cleaner, de-duplicated retrieval returns more relevant results with fewer tokens — and that principle extends to how you serialize those results before they enter the prompt.

The context window is a fixed budget. In a 128k-token window, a 4,000-token system prompt and a 2,000-token answer reserve leave roughly 122,000 tokens for retrieved context. If each chunk consumes 500 tokens as JSON but only 206 tokens as TOON, you fit approximately 592 chunks instead of 244. In practice the gains are more modest because few pipelines push the window to its limit, but the principle is real: the format of your retrieved context determines how many evidence nodes the model can actually see.

Independent research reinforces this. A January 2026 arXiv paper (arXiv 2412.17189, "Talking with Tables for Better LLM Factual Data Interactions") found that providing data in tabular structures yields a 40.29% average performance gain over semi-structured formats such as JSON and knowledge graphs. The study's attention analysis explains why: tables help the model focus on relevant fields more efficiently than nested key-value notation.

TOON's table block is precisely this tabular structure — with the added benefit of declaring field names once in the header rather than repeating them on every row. For the RAG use case, where all retrieved nodes share the same schema, this is close to the ideal encoding. See our full RAG pipeline guide for end-to-end integration patterns.

How Vector DB Results Map to TOON Tables

Results from Pinecone, Weaviate, and pgvector all return a consistent structure: a match id, a relevance score, the chunk text, and a metadata object containing fields like source, date, and document title. This is a textbook uniform array — every node has exactly the same keys.

Below is the same top-3 result set expressed as JSON and as TOON. The JSON version carries each key four times (once per result); the TOON version declares the schema once in the header and reduces each row to its values.

// JSON — repeated keys on every result (~310 tokens for 3 results)
[
  {
    "id": "doc_001",
    "score": 0.94,
    "text": "TOON reduces token overhead by declaring fields once in a header, then listing values row by row.",
    "source": "toonformat.dev",
    "date": "2025-11-01"
  },
  {
    "id": "doc_047",
    "score": 0.91,
    "text": "Uniform arrays are TOON's best-case shape, yielding up to 58.8% fewer tokens versus JSON.",
    "source": "toonformat.dev",
    "date": "2025-11-15"
  },
  {
    "id": "doc_112",
    "score": 0.88,
    "text": "RAG pipelines pass the top-k nodes plus metadata into the LLM context window for grounded answers.",
    "source": "pinecone.io",
    "date": "2024-06-10"
  }
]

// TOON — schema declared once, values only per row (~128 tokens for 3 results, ~59% fewer)
results[3]{id,score,text,source,date}:
  doc_001, 0.94, TOON reduces token overhead by declaring fields once in a header then listing values row by row., toonformat.dev, 2025-11-01
  doc_047, 0.91, Uniform arrays are TOON's best-case shape yielding up to 58.8% fewer tokens versus JSON., toonformat.dev, 2025-11-15
  doc_112, 0.88, RAG pipelines pass the top-k nodes plus metadata into the LLM context window for grounded answers., pinecone.io, 2024-06-10

The header line — results[3]{id,score,text,source,date}: — tells the model the array name, count, and field schema up front. This explicit count and schema declaration is what drives TOON's 99.6% field-retrieval accuracy on uniform data, per the toonformat.dev benchmark (5,016 LLM calls across four models).

For more on the syntax, see the TOON format introduction or the JSON vs TOON token comparison.

JSON vs TOON for Vector DB Results: Token and Accuracy Comparison

The table below uses the official toonformat.dev benchmarks (flat/uniform array data shape) as the basis for the token figures, scaled to a representative RAG result set. Field-retrieval accuracy is the benchmark's direct measurement.

The row count difference is the most operationally significant number. With a fixed context budget allocated to retrieved evidence, switching from JSON to TOON lets you pass more than twice as many chunks to the model. On tasks where recall matters — answering questions over a large document corpus — this directly translates to higher answer coverage.

The token savings also reduce cost. Feeding 20 retrieved chunks to GPT-5 Nano using TOON instead of JSON saves roughly 1,200 input tokens per query. At scale — a thousand queries per day — that compounds quickly. See our API cost optimization guide for a full cost breakdown.

Handling Metadata Fields: What to Include and What to Drop

Most vector databases attach rich metadata to each chunk: source URL, document title, author, publish date, section heading, page number, language, and more. Not all of these are useful in the prompt. Before encoding as TOON, project only the fields the model needs to generate a citation or assess relevance.

A practical projection for a general RAG pipeline is: id, score, text, source, and date. Dropping fields like language or page_number when they are irrelevant to the query saves tokens and reduces noise. The TOON header makes this projection explicit: the field list in results[n]{...}: is the schema the model sees.

One important caveat from the arXiv 2603.03306 study: TOON's efficiency is non-linear. For very small result sets — fewer than about 10 chunks — the format-instruction overhead (teaching the model to read TOON) can approach or exceed the savings. If your top-k is 3 or fewer, plain JSON is simpler and likely cheaper. TOON pays off clearly at top-10 and above.

GraphRAG and Hybrid Search: Where TOON Fits

GraphRAG — the emerging 2026 pattern of combining vector search with a knowledge graph for complex multi-hop reasoning — returns two kinds of data: the vector-retrieved node list (uniform array, TOON-friendly) and the graph edges connecting those nodes (heterogeneous structure, JSON or a dedicated graph notation). The right approach is to encode each part in the appropriate format: TOON for the node table, JSON or plain prose for the edge list.

Hybrid search results (dense + sparse vectors) also come back as uniform arrays and benefit equally from TOON encoding. The schema is identical regardless of the retrieval method; only the scores differ.

Step-by-Step: Encoding Pinecone / Weaviate Results as TOON

The pattern is straightforward regardless of which vector database you use:

// 1. Query your vector DB (Pinecone example)
const results = await index.query({
  vector: queryEmbedding,
  topK: 10,
  includeMetadata: true,
  filter: { date: { $gte: "2024-01-01" } },
});

// 2. Project only the fields you need
const nodes = results.matches.map((m) => ({
  id: m.id,
  score: m.score.toFixed(3),
  text: m.metadata.text,
  source: m.metadata.source,
  date: m.metadata.date,
}));

// 3. Encode as TOON (using @toon-format/toon)
import { encode } from "@toon-format/toon";
const toonContext = encode({ results: nodes });

// 4. Insert into prompt
const prompt = `Answer the question using only the retrieved evidence below.
${toonContext}
Question: ${userQuery}`;

The encode call detects that results is a uniform array and automatically generates the TOON table block. You can convert and inspect results manually using the free json2toon converter before integrating into your pipeline.

For a complete walkthrough of integrating TOON into a LangChain or LlamaIndex retriever, see the companion post on MongoDB documents to LLMs — the same projection-then-encode pattern applies.

Frequently Asked Questions