Claude's New Tokenizer Uses ~30% More Tokens: What to Do

Claude Opus 4.7 and every later Claude model use a newer tokenizer that, according to Anthropic, "produces approximately 30% more tokens for the same text." Per-token prices did not rise to match, so the same prompt can cost more and fill more of the context window. Recount your prompts on the new model, and shrink structured data so there are fewer tokens to pay for.

What changed in Claude's tokenizer?

Anthropic's pricing page states: "Claude 4.7 and later models and Claude Mythos Preview use a newer tokenizer that contributes to their improved performance on a wide range of tasks. This tokenizer produces approximately 30% more tokens for the same text. The exact increase depends on the content and workload shape. Claude Sonnet 4.6 and earlier models use the previous tokenizer."

That covers Opus 4.7, Opus 4.8, Opus 5, Opus 5.5, Sonnet 5, Sonnet 5.5 and the Fable models. The token counting docs add that Fable 5.1, Mythos 5.1, Fable 5 and Mythos 5 "share the tokenizer introduced with Claude Opus 4.7". Claude Haiku 4.5 and Sonnet 4.6 are still on the previous tokenizer.

The change also affects how much text fits in the context window. The models overview says "1M tokens is roughly 555k words or 2.5M Unicode characters on the current tokenizer (introduced with Claude Opus 4.7); models before it fit about 750k words in 1M tokens."

What does 30% more tokens do to my bill?

You pay per token, so cost depends on both the price and the token count. A rough way to compare models across the change is to multiply the new models' prices by about 1.3, which converts them into "old-tokenizer" terms. Prices below are standard-tier input and output rates per million tokens from Anthropic's pricing page, checked October 1, 2026:

Two things stand out. Moving from Opus 4.6 to Opus 4.7 at the same list price was effectively a price increase of about 30% for the same text. And the cheaper 5.x list prices still come out roughly level with, or below, the older models once you account for the tokenizer. The 1.3 factor is only an average. Anthropic says the actual increase "depends on the content and workload shape", so measure your own prompts rather than relying on this table.

How do I measure the change for my prompts?

Use the free token-counting endpoint, POST /v1/messages/count_tokens. It "counts under the tokenizer of the model you pass". The docs suggest counting the same request twice, once with your current model and once with the target model, and comparing the input_tokens values. In Python:

import anthropic

client = anthropic.Anthropic()
messages = [{"role": "user", "content": open("prompt.txt").read()}]

for model in ["claude-sonnet-4-6", "claude-sonnet-5-5"]:
    r = client.messages.count_tokens(model=model, messages=messages)
    print(model, r.input_tokens)

Token counting is free but rate-limited (5,000 requests per minute on the Start tier), and its limits are separate from message creation. Its count is "an estimate" that can differ slightly from what you are billed. Anthropic's migration advice is explicit: "don't reuse token counts measured on the older model to estimate costs or context window fit."

Why do compact data formats matter more now?

A tokenizer change multiplies whatever you send. If 60% of a prompt is JSON tool results or retrieved records, that JSON now costs roughly 30% more to send than it did on Sonnet 4.6. The most direct fix is to send fewer characters. Repeated keys, quotes and braces are the easiest characters to remove.

TOON writes the keys of a uniform array once, in a header. For example, this JSON:

{
  "products": [
    {
      "sku": "A-100",
      "name": "Desk lamp",
      "price": 39.5,
      "stock": 120
    },
    {
      "sku": "A-101",
      "name": "Monitor arm",
      "price": 89,
      "stock": 42
    },
    {
      "sku": "A-102",
      "name": "USB-C hub",
      "price": 24.99,
      "stock": 310
    }
  ]
}

becomes this in TOON (real @toon-format/toon 4.1.1 output):

products[3]{sku,name,price,stock}:
  A-100,Desk lamp,39.5,120
  A-101,Monitor arm,89,42
  A-102,USB-C hub,24.99,310

On the official TOON benchmark, TOON uses about 43% fewer tokens than JSON on average, from about 33% on mixed/nested data to about 59% on flat tables. The current run reports 42.6% fewer tokens with 72.2% accuracy against 71.4% for JSON over 5,856 calls (benchmark results). One caveat: those figures use OpenAI's o200k_base tokenizer, and the TOON README notes that absolute counts depend on the tokenizer. The relative saving should be similar on Claude, but confirm it with count_tokens on your own data before you budget around it.

Expect smaller savings across the whole prompt. In Halodoc's production rollout, savings were 5–15% because only part of each prompt was JSON.

What else can offset the tokenizer increase?

• Prompt caching. Cache hits cost 0.1x the base input price, or 0.05x on Opus 5.5. A large, stable prefix is the biggest saving available. See TOON and prompt caching.
• Batch API. 50% off input and output for work that is not time-sensitive.
• Model choice. Haiku 4.5 is still on the previous tokenizer at $1 / $5 for simple extraction work.
• Trim tool results. Drop fields the model never uses before serializing them.

For background on how tokenizers split text, read how LLM tokenization works and our tokenizer comparison. For Claude-specific integration, see TOON with Anthropic Claude. To measure a payload quickly, paste it into the TOON token calculator.

Frequently Asked Questions