Most LLM code is written as if text were a sequence of characters or words. The model sees neither. It sees tokens, and the mismatch between how you count and how the model counts produces a steady trickle of bugs that are hard to spot because nothing throws an error.

This article covers where the gap bites and the habits that keep it from becoming an incident. The provider-specific details come from Anthropic's token-counting documentation; the principles apply to any provider.

Don't use string length as a budget

A limit like "4,000 characters" or "800 words" is a proxy for a token budget, and a poor one. Token density depends on content. Ordinary English prose is relatively cheap per character. Source code, JSON, URLs and encoded blobs are usually much more expensive, and text in some non-English languages and scripts tends to cost noticeably more tokens for the same number of characters. A limit that's safe on your English test set can overflow on real input. Measure with your own data and don't rely on rules of thumb.

The tokenizer changes between models

This one is documented, and it's easy to miss. Anthropic's documentation says its newer models, starting with Claude 4.7, use a newer tokenizer, and that the same input text produces approximately 30 percent more tokens than on earlier models, with the exact increase depending on content. Its advice is to recount prompts against the model you plan to use instead of reusing counts from an older one, because billing and context-window fit both follow the new count.

The lesson generalizes. A token budget is tied to a specific model. Swapping models without recounting can push you past a limit you believed you had margin on, and raise your bill without any change to your prompts.

Use the provider's counter

You don't have to guess. Anthropic offers a token-counting endpoint that accepts the same structure as a message request, including system prompts, tools, images and PDFs, and returns the input token count. Points from the docs worth knowing:

  • It's free to use, subject to requests-per-minute rate limits by usage tier, separate from message-creation limits.
  • The result is an estimate, and the actual count can differ by a small amount.
  • It doesn't use caching logic. Cache markers in a counting request have no effect, because caching happens only when you create a real message.
  • Some inputs, such as certain server tools and the MCP connector, return an error instead of a count. For those, read the usage object in the real response.

Whichever provider you use, count with its tokenizer or counting endpoint and not with len(). Use an approximation only for rough dashboards, with a safety margin.

Formatting is not free

Whitespace, indentation and repeated newlines all consume tokens. Pretty-printed, deeply nested JSON can cost noticeably more than the compact form, with no change in meaning. When you're paying per token or short of context, trim what the model doesn't need.

Identifiers: don't ask the model to retype them

Long numbers, UUIDs and hashes are typically split into several arbitrary chunks. That's one reason models are unreliable at character-level work such as counting letters, reversing strings or copying a long ID exactly. Anthropic's guide on writing tools for agents makes a related point: resolving arbitrary alphanumeric IDs into meaningful language significantly improves precision. If a feature depends on an identifier surviving a round trip through the model, pass a short handle and resolve it in your own code. See Designing Tool Error Messages an Agent Can Act On for more on tool design.

Habits that help

  1. Count at the boundary. Enforce the budget where text enters your pipeline, and decide in advance what gets dropped: oldest turns, lowest-ranked retrieved chunks, or an explicit error. For conversations, see Multi-Turn State.
  2. Recount after any model change. Treat it as part of the upgrade checklist in Silent Model Upgrades Are a Production Risk.
  3. Test with ugly inputs. Include a code-heavy sample, a non-English sample and a long-identifier sample in your evaluation set.
  4. Keep the model out of character-level work. Counting, exact copying and string manipulation belong in ordinary code that the model calls as a tool.
  5. Watch for truncation. A budget that's too tight shows up as cut-off output. See Truncated Output Is a Silent Failure.

The takeaway

"How many tokens is this, on this model?" is a question your code should answer explicitly, not infer from string length. The fix is rarely complicated. You only apply it once you stop assuming the model reads text the way you do.

Sources

Developer ToolsSoftware Engineering