TOON in Production: Lessons From Halodoc's 5–15% Savings

Health-tech company Halodoc moved production LLM workloads from JSON to TOON input and cut cost by about 5% to 15% per use case, with p95 latency 1% to 12% lower. That is far below the 35–45% token reduction they estimated in theory. The reason is simple: only part of each prompt was JSON. Their write-up is one of the few public production reports on TOON.

What did Halodoc actually measure?

According to the Halodoc engineering post, the team migrated several AI agents and direct LLM API calls to send TOON instead of JSON. For each use case they measured input token reduction, output token reduction (in a proof of concept), cost savings, p95 latency change and response quality. They compared responses to TOON input against a JSON-input baseline on the same test data. Classification flows were scored on precision and recall, and generation flows were reviewed by hand.

Production use cases ran on "GPT-4.1, GPT-5-mini, GPT-4o, Amazon Nova Pro, Amazon Nova Lite, and a GPT-OSS-120B model on Bedrock". They only migrated use cases where TOON matched or beat baseline accuracy. The post also cites outside compatibility testing on Claude Haiku 4.5, Gemini 2.5 Flash, GPT-5-nano and Grok-4, reporting about 95% LLM compatibility across providers. That is a useful signal, but it does not replace testing on the model you actually run. Halodoc also notes that TOON "is not yet widely adopted as an industry standard", which is why they invested in fallbacks and debugging aids.

Why were real savings so much lower than the benchmarks?

Halodoc explains it directly: "Theoretical savings of 35-45% assume all content is JSON. In practice, most prompts mix structured data with natural language instructions, so real savings will be lower."

The math is straightforward. Suppose a prompt is 30% JSON and 70% instructions, examples and conversation. Even a 45% cut in the JSON part reduces the whole prompt by only about 13.5%. The official TOON benchmark reports 42.6% fewer tokens than JSON (retrieval-accuracy results), but it measures the data alone. Your bill covers the whole request, including system prompt, tool definitions and output tokens, and TOON changes none of those.

The use case with the best result shows this. Halodoc writes that "the eligibility classification workflow showed 13.8% savings because its prompt includes large JSON payloads of eligibility rules and item details." When the structured part of the prompt grows, the saving grows with it.

What kind of payload benefits most?

Large arrays of uniform records benefit most. Here is an illustrative payload in the style of an eligibility check (our example, not Halodoc's data):

{
  "member": {
    "id": "M-88231",
    "plan": "gold"
  },
  "claims": [
    {
      "code": "C001",
      "category": "consultation",
      "amount": 150000,
      "covered": true
    },
    {
      "code": "C002",
      "category": "lab",
      "amount": 320000,
      "covered": false
    },
    {
      "code": "C003",
      "category": "pharmacy",
      "amount": 85000,
      "covered": true
    }
  ]
}

In TOON (real @toon-format/toon 4.1.1 output) the claims array becomes a table and each key appears once:

member:
  id: M-88231
  plan: gold
claims[3]{code,category,amount,covered}:
  C001,consultation,150000,true
  C002,lab,320000,false
  C003,pharmacy,85000,true

With three rows the saving is modest. With hundreds of rows it approaches the official flat-data figure of 58.7% fewer tokens than JSON (token-efficiency results). A single nested config object gains far less. The TOON README says compact JSON often wins on deeply nested or non-uniform data.

How did Halodoc roll TOON out safely?

Their engineering choices are worth copying even more than their numbers:

• Per-use-case flags. "TOON is enabled via per-use-case configuration flags (separate for input and output)." Their advice is to "use config flags, not hard switches".
• Fail back to JSON. "If the converter throws an exception, the system catches it and falls back to standard JSON serialisation."
• Keep JSON for debugging. They log "a JSON-equivalent alongside the TOON payload" and store all response data in JSON, because TOON lacks standard tooling.
• Staged promotion. Enable TOON on staging, validate accuracy against the JSON baseline, then promote to production.
• Output stays experimental. TOON output reached only the proof-of-concept stage, which fits the 2026 research showing models write TOON less reliably than they read it.

That last point matches the agent research. See what the 2026 agent study found about models writing TOON.

How do I run my own TOON migration test?

1. Find structured-heavy, high-volume calls. Rank endpoints by monthly input tokens multiplied by the share of each prompt that is JSON. Halodoc's first recommendation is to "start with high-volume, structured-data-heavy use cases".
2. Split the prompt. Keep instructions as text and convert only the data block. Halodoc's advice is to "split your prompts".
3. Measure on real traffic. Replay a sample of production requests in both formats and count tokens with your provider's tokenizer, not an estimate. In Halodoc's words: "measure on real data, not theoretical estimates."
4. Gate on quality. Use the metric that already matters for that flow, such as precision and recall or a review rubric, and migrate only if TOON matches the baseline.
5. Ship behind a flag with a fallback. Fall back to JSON on any encoding error and log the JSON equivalent.

You can test step 3 by hand: paste a real payload into the TOON token calculator to see the token difference. For more on cost modeling, see the LLM cost calculator guide. For the cases where JSON is still the better choice, read when not to use TOON.

What is the takeaway?

A 5–15% saving is less than the headline figures, but it is real and sustained, and it came with lower latency at no loss of accuracy. For a high-volume LLM service that can be worth the effort. Set expectations from the share of each prompt that is structured data, not from benchmark headlines.

Frequently Asked Questions