Tokenization traps in real apps

Token counts affect context limits, latency and cost, but they are not the only factors. A token is a unit in the model’s vocabulary, not necessarily a whole word or character. Count with the tokenizer and revision used by your deployment, and distinguish a local text estimate from usage for the complete server request.

Record the model, tokenizer and request format used by the application. Use representative inputs when checking token estimates and observed usage.

Try Compute: Launch a vLLM endpoint on Compute. Pick the model you use in production and measure input and output token counts with streaming on. Keep the endpoint in the chosen region and set an appropriate output limit for each route.

Introduction to Tokenization

Tokenization sits at the heart of how language models work. When you send text to a model, it can't read words the way you do—it needs to break your input into smaller pieces called tokens first. Think of tokenization as the bridge between human language and what the machine can actually process. These tokens might be whole words, parts of words, or even single characters, depending on how the system is set up.

BPE and WordPiece are tokenization algorithms. SentencePiece is a toolkit that supports both BPE and unigram models, not a separate algorithm opposed to BPE. Vocabulary, normalization and special-token rules also affect the result; use the tokenizer supplied for the pretrained model.

Understanding how text becomes tokens helps you investigate unexpected counts and set request budgets. It does not by itself guarantee better prompts or the intended answer.

Why token counts drift

• Different tokenizers. Two tokenizers can produce different counts even when both use BPE. Record the model, tokenizer revision, vocabulary and configuration instead of inferring them from a model-family name.
• Invisible characters. NBSPs, zero‑width joiners, smart quotes, and mixed line endings change counts.
• Prompt assembly. System prompts and examples get duplicated or reformatted by clients, and dependencies between prompt components can affect token counts.
• Request serialization. Message roles, tool definitions and other request structure may contribute tokens. Rendering can happen in the client or server; a count of visible prompt text alone does not cover the complete request.
• Language quirks. Compounds (German/Dutch), diacritics (French/Spanish), clitics (Arabic/Spanish), CJK segmentation—all shift counts.

Common pitfalls and quick fixes

Model and SDK differences (plain English)

• Check the specific model generation. Llama 3 uses a tiktoken-based BPE tokenizer; “GPT uses BPE, Llama uses SentencePiece” is not a reliable rule. Consult the model’s tokenizer files and configuration.
• Chat templates. Use the template expected by the deployed model. If you render a template locally, avoid adding its special tokens a second time. Do not assume a raw prompt and structured messages are equivalent inputs.
• Stop sequences. Where the API supports them, stop sequences apply to generated output. Their presence in the prompt alone does not trigger a stop. Check support for the specific model and endpoint.

Check preprocessing before counting

1. Preserve the original line endings unless your application deliberately converts them. Count the text that will actually be sent.
2. Do not strip ZWJ/ZWNJ or replace NBSP as a blanket token-saving step. These characters can carry linguistic or formatting meaning.
3. Preserve repeated spaces and tabs in code, structured data and quoted material. Test any deliberate whitespace transformation against those cases.
4. Do not convert quotes and apostrophes to ASCII merely to reduce token counts. Keep the intended language and exact source text where required.
5. Keep Markdown fences and meaningful trailing whitespace. Any removal must be an intentional application transformation, not an automatic counting shortcut.
6. Canonicalise system prompts (single source; versioned).
7. Set the target output language in the system prompt.
8. Log the server’s input and output usage categories when available, alongside the local estimate and model revision. Do not log sensitive prompt content unnecessarily.

Make preprocessing an explicit, tested application choice. Unicode normalization can change representation; compatibility normalization can remove distinctions. Counting alone is not a reason to rewrite each sentence.

Test on a compatible instance

Check the available instance configuration and current product documentation before starting a test. Measure the behavior of your own model and workload.

Try Compute

A small testing harness (compare tokenizers)

Keep a tokenizer baseline for each model and revision. The sketches below compare local plain-text counts; they do not count a complete chat request or demonstrate equivalent behavior across models. Add representative regression tests before relying on them in CI.

Scope of these sketches: Python explicitly loads cl100k_base and a local SentencePiece file named llama.model; that file is required and is not a tokenizer for every Llama generation. The JavaScript sketch selects gpt-4o-mini, whose documented encoding is o200k_base, and uses different samples. These are not a matched Python/JavaScript comparison. Align the tokenizer and inputs before comparing results; neither sketch was runtime-tested for this article.

Python (sketch)

# pip install tiktoken sentencepiece
import tiktoken
import sentencepiece as spm

# Loading the tokenizer: make sure to load the correct encoding or model for your use case.
enc_gpt = tiktoken.get_encoding("cl100k_base")
sp = spm.SentencePieceProcessor(model_file="llama.model") # use your model's spm

# Each entry in TEXTS is an instance or example sentence to be tokenized.
TEXTS = [
"l’été dernier à Zürich — café & résumé",
"Schadenfreude und Lebensversicherung",
"اللغة العربية جميلة جدًا",
"I typed :shrug: and got 🤷🏽‍♂️",
"""```


print('big code block that should be truncated...')
```"""
]

for s in TEXTS:
# The text is tokenized using both libraries/tools.
gpt_tokens = len(enc_gpt.encode(s))
sp_tokens = len(sp.encode(s))
print({"text": s[:32] + "…", "gpt": gpt_tokens, "spm": sp_tokens})


Node (rough sketch)

// npm i js-tiktoken
import { encodingForModel } from "js-tiktoken";
// Loading the tokenizer for the desired model.
const enc = encodingForModel("gpt-4o-mini");
// Each sample is an instance or example sentence.
const samples = [
"l’été à Paris",
"Schweizer Rückversicherung",
"emoji 🤷🏽‍♂️",
];
for (const s of samples) {
// Output how many tokens each sentence is tokenized into.
console.log(s.slice(0,24)+"…", enc.encode(s).length);
}


Context-Aware Development

Building effective LLM applications isn't just about feeding text into a model—it's about understanding context and how it shapes the quality and speed of what you get back. Context-aware development means designing systems that use your model's strengths while managing the real challenges of token limits and context windows.

RAG adds retrieved material to the input and can help when that material is relevant. It does not guarantee a smaller prompt, lower latency or a more accurate answer. Measure retrieval quality, context length, response quality and latency together on representative requests.

When you understand how context, tokens, and model performance work together, you can design applications that get the most from LLMs. This means carefully choosing your input data, watching token usage, and using context-aware techniques so every token delivers value for your specific task. Context-aware development comes down to finding the right balance—give the model enough information to understand what you need while keeping things efficient for real-world use.

Monitoring and alerts

• Compare like-for-like usage. Keep local estimates separate from server-reported request usage. Apply the provider’s billing definitions rather than treating a count of visible text as a bill.
• Calibrate drift alerts. A 5–10% difference over an hour can be an initial diagnostic threshold, not a universal standard. Set thresholds from observed request types and investigate persistent changes.
• Route dashboards. Watch token distributions per route; adjust caps when tails grow.
• Language mix. Track language by requests; big shifts can change counts and latency.
• Incidents. When caps trip or OOMs rise, check for a tokenizer or normalisation change first.

Note: It is critical to monitor how many tokens are used in each request to avoid budget overruns and unexpected costs.

Try Compute: Run a vLLM endpoint on Compute with input and output usage logs, TTFT, and TPS. Record the tokenizer and its revision for each deployed model, then check estimates in CI.

Compare token estimates with observed usage

Track tokenizer and template revisions, deliberate preprocessing and server usage. Retest representative languages and tasks after changes. These checks help explain count drift; they do not guarantee stable time to first token (TTFT) or tokens per second.

Conclusion and Future Directions

For a pretrained model, use its compatible tokenizer rather than independently choosing BPE, WordPiece or a SentencePiece model. Evaluate model quality, latency and cost together before changing the deployment.

When evaluating a tokenizer or model change, compare the same representative inputs and keep the version information with the results. Use measured differences rather than assuming that a newer option is more efficient.

Evaluate context-management changes on your own tasks. More retrieved material or a newer tokenizer does not by itself establish better answers; retain measured comparisons and the exact model configuration.

Keep these checks in the application’s test process and repeat them when the model, input handling or request format changes.

FAQ

What’s the safest way to count tokens?

Use the tokenizer and template for the deployed model and revision. Compare local estimates with the provider’s request-level counting or usage information where available.

Separate estimated tokens, reported usage and billed cost. Investigate differences before using a local estimate for billing or alerts.

For supported models, tiktoken can estimate plain-text tokens locally. Counting a string alone does not include the complete structured request.

Why are counts different between OpenAI and Llama‑family models?

Models can use different vocabularies, normalization rules and special tokens, including when they share a tokenization algorithm. SentencePiece can implement BPE; it is not an opposing algorithm.

To compare a sentence, run the exact same text through each named tokenizer and revision and record the token IDs and count. Do not infer a token split from the displayed words or present an unmeasured segmentation as a result.

Note: Always plan for variance in token counts between models, and set token caps accordingly.

Do Unicode characters really matter?

Yes. Whitespace, combining characters, emoji sequences and punctuation can affect the result, but they do not always increase it. Test the exact Unicode string with the deployment’s tokenizer.

Count after any intentional application preprocessing, and keep the original text for comparison where appropriate. Do not normalize solely to force a lower count.

To explain, Unicode characters can split words or sentences into additional tokens because tokenizers may treat them differently from standard spaces or punctuation. This can affect how sentences are processed and the total token count.

Can I use word counts instead of token counts?

No. A word count is not a token count, and the relationship varies with language, text and tokenizer. Measure the actual input instead of applying a fixed words-to-tokens conversion.

To explain, word counts are misleading because tokenizers may divide a single word into multiple tokens or merge parts of different words, depending on the language model. This affects both cost and latency calculations.

Note: Always use token counts instead of word or sentence counts when estimating usage or costs.

How do I budget for multilingual apps?

Log language tags per request to accurately track usage and manage token allocation. Different tasks, such as translation, summarization, or dialogue generation, may require different budgeting strategies depending on their complexity and language. Measure token count for each request to determine how many tokens are used, ensuring you stay within model context limits.

Set multilingual budgets from measured input and output distributions for each task and language, including longer cases. A multilingual tokenizer label does not guarantee equal token efficiency or answer quality across languages.

Measure TTFT and TPS by language.

Does streaming change token counts?

Streaming changes how output is delivered; it is not a token discount. Compare usage for the actual completed or canceled request, not the amount of text displayed so far.

A Stop button should request cancellation through the supported client/server path. Hiding output or closing the display does not prove generation has stopped; cancellation can race with work already in progress. Do not promise that every token after the click is unbilled.

For OpenAI Chat Completions, requesting stream_options.include_usage adds a final usage chunk. An interrupted stream may not deliver it. Record that usage is unavailable rather than treating a missing total as zero.

Use the output-limit parameter supported by the endpoint and model, such as max_output_tokens or max_completion_tokens where applicable. Allow for non-visible output categories when relevant; test truncation and cancellation behavior before setting tight limits.

‍