Optimize OpenAI and Claude API Costs with TOON

In the SaaS world, we are used to fixed costs. You buy a server, you pay for the server. You buy a database, you pay for storage. But Generative AI introduces a new, variable cost model: Token-Based Pricing.

This model is deceptive. A prototype that costs pennies to run during development can suddenly bankrupt a startup when it hits 10,000 users. Why? Because every single interaction scales your costs linearly with the verbosity of your data.

In this guide, we will explore the Unit Economics of AI and provide 6 concrete strategies to slash your API bills by up to 80% without sacrificing intelligence.

The Silent Killer: Verbosity

Let's understand the enemy.

Suppose you are building a "Chat with your Data" app. A user asks: "What were my sales last month?"

To answer, your app might retrieve a list of 500 transactions from the database and feed them into a mid-tier model such as Claude Sonnet 5.5 or GPT-5.6 Terra. Both list input at $2.00 per 1M tokens as of October 2026 (Anthropic pricing, OpenAI pricing).

• JSON Format: 10,000 tokens. Input cost: $0.020.
• TOON Format: about 4,130 tokens (58.7% fewer, the flat-data figure from the official TOON benchmark). Input cost: about $0.008.

"It's barely a cent," you say.

Now multiply by 1,000 users asking 5 questions a day.

• JSON: 5,000 calls × $0.020 = $100/day = $3,000/month.
• TOON: 5,000 calls × $0.00826 = about $41/day = about $1,240/month.

That is roughly a $21,000 annual difference on input tokens alone, for a single endpoint, and it scales linearly with users, context size, and model tier. Run the same workload on a frontier model such as Claude Opus 5.5 ($4 per 1M input) and the gap doubles. Profitability in AI is an engineering constraint.

Strategy 1: Format Optimization (The Low Hanging Fruit)

This is the easiest win because it requires zero changes to your model or prompts. You simply change how you serialize data in the context window.

The "Repeating Key" Tax

JSON repeats keys for every object.

{"id": 1, "status": "pending", "category": "sales"},
{"id": 2, "status": "active", "category": "sales"}

TOON declares headers once.

transactions[2]{id,status,category}:
  1,pending,sales
  2,active,sales

In the current official benchmark (v4.1, 5,856 LLM calls across four models), TOON used 42.6% fewer tokens than JSON across all datasets while scoring 72.2% retrieval accuracy versus JSON's 71.4% (toonformat.dev). Savings are largest on flat, uniform arrays and smallest on deeply nested data.

Action Item: implement a middleware that converts all JSON context data to TOON before sending it to the LLM.

Before you optimize prompt payloads, validate them. Use our JSON Schema validator to catch contract errors and our JSON repair tool to fix malformed payloads.

Strategy 2: Semantic Caching

The most expensive API call is the one you make twice.

Users often ask the same questions. "How do I reset my password?" "What is the pricing?"

Traditional Caching (Exact Match): Only works if the string is identical. "Reset password" != "Change password".

Semantic Caching:1. User asks a question.
2. Calculate the Embedding Vector of the question (Cheap).
3. Check your Vector DB for similar questions (Distance < 0.1).
4. If found, return the cached answer.

Impact: You can deflect 30-40% of traffic from your expensive LLM to your cheap Vector DB.

Pair it with provider-side prompt caching for the requests that still reach the model. As of October 2026, a cache hit costs 10% of the base input price on most models from both Anthropic and OpenAI (Claude Opus 5.5 drops to 5%). Caching discounts the price per token; TOON reduces the number of tokens, so the two stack.

Strategy 3: Model Cascading (The Router Pattern)

Not every question requires a frontier model.

"Hello" requires an intern (GPT-5.6 Luna at $0.20 / $1.20, or Claude Haiku 4.5 at $1 / $5 per 1M input / output tokens). "Write a complex legal contract" requires a partner (GPT-5.6 Terra at $2 / $12, Claude Sonnet 5.5 at $2 / $10, or Claude Opus 5.5 at $4 / $20 for the hardest cases). Prices are list prices as of October 2026.

Implementation:Build a "Router" step.

const complexity = classifyComplexity(userQuery); // Uses a fast, small model

if (complexity === "low") {
  return callModel("gpt-5.6-luna");   // or "claude-haiku-4-5"
} else {
  return callModel("gpt-5.6-terra");  // or "claude-sonnet-5-5"
}

Impact: If 80% of queries go to gpt-5.6-luna ($0.20 per 1M input) and 20% to gpt-5.6-terra ($2.00), the blended input price is 0.8 × $0.20 + 0.2 × $2.00 = $0.56 per 1M tokens, 72% below sending everything to Terra. The same split between claude-haiku-4-5 ($1) and claude-sonnet-5-5 ($2) gives $1.20, a 40% cut. Measure the classifier's misroute rate, because a wrong answer from the cheap model can cost more than the tokens you saved.

Strategy 4: Prompt Compression

Prompt Engineering isn't just about quality; it's about brevity.

Verbose Prompt:"Please act as a helpful assistant. I would like you to look at the data below and please, if you would be so kind, summarize it for me in a nice format."

Optimized Prompt:"Summarize the data below."

Models don't need politeness. They need clarity. Removing "Chatty" instructions saves tokens.

Advanced Technique: Use Symbolic Instructions. Instead of saying "Output the result as a list of bullet points," say "Output: Markdown List."

Strategy 5: Fine-Tuning vs RAG

RAG (Retrieval-Augmented Generation): You paste examples and knowledge into every single prompt. You pay for that knowledge every time.

Fine-Tuning: You bake the knowledge into the model weights once.

If you have a static knowledge base (e.g., "Company Policy"), Fine-Tuning a small open-weight model (for example, one from Meta's Llama 4 family or a comparable Qwen or DeepSeek model) can be cheaper in the long run than paying for RAG context on every call.

The Break-Even Point: Usually around 10k-50k requests. If you query the same static data more than that, consider Fine-Tuning.

Strategy 6: The "Output Token" Trap

Output tokens are typically 5–6x more expensive than input tokens. As of October 2026, Claude Sonnet 5.5 lists $2 input vs $10 output per 1M tokens (5x), and GPT-5.6 Terra lists $2 vs $12 (6x).

If you ask a model to "Think step by step," it generates 500 tokens of reasoning before giving the answer. This improves quality but can multiply the cost of the response several times over.

Tip: only ask for reasoning when necessary. Or, ask for the answer first, and only ask for reasoning if the user clicks "Explain Why."

Conclusion

Optimizing API costs is a multi-layered game.

1. Layer 1 (Data): Use TOON to compress the payload.
2. Layer 2 (Prompt): Be concise.
3. Layer 3 (Architecture): Use Caching and Routing.

By applying these strategies, you turn AI from a "Cost Center" into a sustainable, scalable technology.