Tokenization — llm-kit

tokenization

// AI can't read words — it reads numbers. Tokenization converts text to numbers.

💡 The core idea: Computers can only work with numbers, not words. So before an AI can read your message, it splits the text into chunks called "tokens" and gives each one a number. "Hello world" might become [15339, 995].

TRY THESE EXAMPLES:

What is a token?

A token is a small piece of text — usually a word or part of a word. AI models don't read full words — they read tokens. "tokenization" might become ["token", "ization"] — two tokens. Common words like "the" are usually one token. Rare or long words get split into smaller pieces. The AI turns each token into a number to process it.

BEGINNER FRIENDLY