AI tokens vs words: count a request before estimating cost

Separate words, Unicode characters and tokens

A visible word may become one token, several smaller tokens or a combination that includes punctuation and spaces. Hugging Face’s tokenizer introduction explains word, character and subword approaches. Language, spelling, code, whitespace and tokenizer vocabulary can all affect the split.

A Unicode character count answers another question. Some characters need multiple code units or bytes, and a visible symbol can contain several Unicode code points. None of these quantities has a universal conversion to model tokens. Treat a characters-per-token rule as a rough heuristic for a stated sample, not a property of all languages.

Count a local text sample with a known tokenizer

For a tokenizer that uses the Hugging Face tokenizers format, load the correct local tokenizer.json and encode a UTF-8 text file. Record the model or tokenizer identifier, tokenizer-file revision, package version and whether special tokens are included. The example targets the published tokenizers 0.23.2 API, disables padding and truncation, and counts raw text without adding special tokens.

Use a tokenizer file obtained from the intended model’s official distribution. This example does not download a model, send text to a server or reproduce a hosted chat API’s full request format. The count becomes useful only when the tokenizer and settings match the specific question you are measuring.

# Example environment: tokenizers==0.23.2
from pathlib import Path
from importlib.metadata import version
from tokenizers import Tokenizer

# Use the matching tokenizer.json already saved locally.
tokenizer = Tokenizer.from_file("tokenizer.json")
tokenizer.no_padding()
tokenizer.no_truncation()
text = Path("sample.txt").read_text(encoding="utf-8")
encoded = tokenizer.encode(text, add_special_tokens=False)
print("tokenizers version:", version("tokenizers"))
print("raw-text tokens:", len(encoded.ids))
# Record tokenizer identity/revision separately.
# This is not a complete chat-request or billing count.

Include the entire request when planning context

A chat request can include system instructions, conversation history, retrieved passages, tool definitions and formatting in addition to the latest user message. Chat templates can insert control tokens around roles and message boundaries. Counting only the last pasted paragraph can therefore understate the input.

For a hosted provider, use its documented request-counting facility when available and supported for your input types. Claude’s token-counting documentation describes structured input counting and notes cases that need actual usage reporting instead. Treat any preflight result according to that provider’s stated estimation limits, especially for tools, images and documents.

Keep credentials and sensitive customer text out of unknown online counters. A local tokenizer can help with raw text, while a provider endpoint sends the supplied request to that provider. Decide which workflow is appropriate for the data before measuring it.

Build a reproducible comparison worksheet

Choose a few representative samples: a normal support message, a code snippet, a table and a passage in each language your application uses. Preserve the exact input bytes so the comparison does not accidentally change line endings, punctuation or whitespace between runs.

For each sample, record the tokenizer identifier and revision, special-token setting, raw-text count and full-request estimate when available. Leave a value unmeasured rather than substituting a guessed ratio. If two tokenizers produce different counts, that is a measured difference for these samples, not a ranking of their language quality.

Reserve output space before adding more input

For a simplified planning example, assume your chosen model permits a combined 32,000-token context for the request type you use. Reserve 4,000 for output and 2,000 as a planning buffer. That leaves 26,000 for measured input. If instructions, history and tools use 8,000 of that budget, 18,000 remains for retrieved content.

These are explicit hypothetical numbers, not a current model specification. Providers can define context, output, reasoning and tool usage differently. Check the current model contract, then enter your actual quantities into the context-budget calculator. A request fitting the context limit does not imply that every input passage is useful.

Hypothetical planning budget:
32,000 combined context
− 4,000 output reserve
− 2,000 planning buffer
= 26,000 input budget

26,000 − 8,000 instructions/history/tools
= 18,000 available for retrieved content

Compare estimates with actual usage and cost

After a small real test, save the provider’s returned usage fields and compare them with your estimate for the same request. Account for any documented cached input, output, reasoning or other billable categories separately. An estimate that omitted a tool definition cannot be reconciled by changing the word-to-token ratio.

Use the AI token-cost calculator with actual token counts and the current prices you provide. Keep input and output separate and retain the price units, such as cost per million tokens. A useful cost worksheet is reproducible from measured usage and stated rates; it does not depend on a universal conversion from words.