AI Token Counter

Tokens are the fundamental units that large language models use to process text. A token can be a whole word, a subword, a single character, or even a punctuation mark, depending on the model's tokenizer. For English text, one token is roughly 4 characters or about 0.75 words. For CJK languages (Korean, Chinese, Japanese), each character typically consumes 1-2 tokens because these characters are less common in the training data. Understanding token counts is essential because LLM APIs charge per token and enforce context window limits.

In English, 1,000 words typically translates to approximately 1,300-1,500 tokens, depending on word complexity and the specific tokenizer. Common short words like "the," "is," and "a" are usually single tokens, while longer or less common words may be split into 2-3 subword tokens. Code, technical terminology, and URLs tend to produce more tokens per word than everyday prose.

Each LLM provider uses a different tokenizer with its own vocabulary. OpenAI's GPT models use tiktoken (cl100k_base or o200k_base), Anthropic's Claude uses its own BPE tokenizer, and Google's Gemini uses SentencePiece. These tokenizers were trained on different datasets and have different vocabulary sizes, so the same text can produce different token counts. This tool provides estimates based on general tokenization rules, which are accurate enough for cost planning and prompt optimization.

Several strategies can reduce token usage: (1) Write concise prompts by removing unnecessary instructions and examples. (2) Use shorter variable names and abbreviations in few-shot examples. (3) For multilingual applications, note that English typically uses fewer tokens than CJK languages for the same meaning. (4) Implement prompt caching where supported (Anthropic's prompt caching, OpenAI's cached completions). (5) Use system prompts efficiently since they count toward every request. (6) Consider using smaller, cheaper models for simpler tasks.

The context window is the maximum number of tokens an LLM can process in a single request, including both the input prompt and the generated output. GPT-4o supports up to 128K tokens, Claude 3.5 Sonnet supports 200K tokens, and Gemini 1.5 Pro supports up to 2M tokens. If your input exceeds the context window, the model will either truncate your input or return an error. This token counter helps you verify that your prompts fit within these limits before making API calls.

GPT-4o pricing is approximately $2.50 per 1 million input tokens and $10.00 per 1 million output tokens. Claude Sonnet is approximately $3.00 per 1M input tokens and $15.00 per 1M output tokens. Gemini Pro is approximately $1.25 per 1M input tokens and $5.00 per 1M output tokens. This tool calculates estimated input costs based on these rates, helping you compare providers and budget for your AI applications. Note that pricing may change, so always verify current rates on each provider's pricing page.

LLM tokenizers are trained primarily on English-dominant datasets, so English words are well-represented in the tokenizer vocabulary and compress efficiently (about 4 characters per token). Korean, Chinese, and Japanese characters appear less frequently in training data, so they are often encoded as individual tokens or even split into multiple byte-level tokens. A Korean sentence expressing the same idea as an English sentence can use 1.5 to 2 times as many tokens. This tool applies a CJK-aware heuristic (1 token per 2 characters when CJK ratio exceeds 30%) to provide more accurate estimates.

This tool provides estimates based on general tokenization heuristics rather than the exact tokenizer algorithms used by each model. For precise counts, you would need to use each provider's official tokenizer library (tiktoken for OpenAI, etc.). However, this tool's estimates are accurate enough for prompt planning, cost budgeting, and context window management. The CJK-aware adjustment makes it particularly useful for multilingual applications where standard English-only estimators significantly undercount tokens.