Safeguard
Supply Chain Security•

How LLMs Read Text: Tokenization, Embeddings and Positional Encoding

How LLMs turn text into numbers: BPE, WordPiece, SentencePiece, tiktoken, embeddings, and sinusoidal, RoPE and ALiBi positions, from the papers.

Safeguard Security Team
65 min read

A large language model never reads letters. Before any attention layer runs, a tokenizer cuts your text into subword pieces and maps each to an integer ID, an embedding matrix turns each ID into a vector of a few thousand numbers, and a positional scheme tells the model where each vector sits in the sequence. Those three steps decide what the model can see, how much text fits in its context window, what it costs to run, and a surprising number of its odd failures.

This post walks through all three, starting from the original papers: byte pair encoding (Sennrich et al., 2016), WordPiece, SentencePiece and the unigram language model (Kudo, 2018), GPT-2's byte-level BPE and OpenAI's tiktoken, word2vec (2013) and the embedding matrix, then positional encoding from the sinusoids of "Attention Is All You Need" (2017) through learned positions, RoPE (Su et al., 2021) and ALiBi (Press et al., 2021). Every algorithm gets a worked toy example you can check by hand.

It is written for engineers, security people and technical leaders who use LLMs and want to know what actually happens to their input. You do not need a maths background. Every symbol is defined where it appears, and every formula is in plain text. It ends with the documented security risks that live at this layer and a short note on securing what you build on top.

This is post 2 of a 10-part series. Post 1, The transformer, explained from "Attention Is All You Need", covers the attention layers that consume everything described here.

Contents

What happens to text before an LLM sees it?

Text goes through four steps before the first attention layer. A tokenizer splits it into subword tokens and maps each to an integer ID. An embedding matrix swaps each ID for a learned vector. A positional scheme adds order information. Only then do the transformer blocks run, and at the end the model predicts a probability for every token ID in the vocabulary.

Take the sentence "The cat sat." and run it through the tokenizer OpenAI uses for GPT-4, called cl100k_base. You get four integers: 791, 8415, 7731 and 13. "The" is 791, " cat" (with its leading space) is 8415, " sat" is 7731 and the full stop is 13. The model will never see the letters c, a and t. It sees the number 8415.

Each of those integers then selects one row of a large table called the embedding matrix. If the model's internal width (called d_model) is 4,096, each token becomes a list of 4,096 numbers, so the sentence becomes a grid of 4 rows and 4,096 columns. Position information is either added to that grid once or injected inside every attention layer, depending on the scheme. The transformer blocks then transform the grid layer by layer, and the final layer projects each row back to a score for every one of the 100,277 IDs in the vocabulary.

The LLM input pipeline in eight steps: raw text, regex pre-tokenization, subword merging, token IDs 791 8415 7731 13, embedding lookup to a 4 by d_model matrix, adding position information, transformer blocks, and output logits over the vocabulary

Figure 1. The input pipeline, with the real cl100k_base IDs for "The cat sat." Steps 1 to 4 are the tokenizer, a separate program. Steps 5 and 6 are the first layers of the model and the subject of the second half of this post.

One detail matters for everything that follows: the tokenizer is trained separately, before the model, usually on a different and smaller sample of text. It is then frozen. The model learns to live with whatever pieces the tokenizer hands it. Several of the quirks and security issues later in this post come straight from that split.

Why do language models need tokenization at all?

A neural network can only learn a vector for symbols in a fixed list. Whole words make that list unbounded, so new or rare words become "unknown". Single characters or bytes keep the list tiny but make every sequence several times longer, and attention cost grows with sequence length. Subword tokenization sits between the two: common words stay whole and rare words are spelled from pieces.

Sennrich, Haddow and Birch put the problem precisely in 2015. Neural translation models of the time had vocabularies typically limited to 30,000 to 50,000 words, yet translation is "an open-vocabulary problem". Names, compounds, numbers and new words never stop arriving. A word-level model has two bad options for a word outside its list: emit an [UNK] placeholder, or fall back to a dictionary lookup bolted on after the network. Both break on German compounds, inflected Russian, or a product name invented last week.

The obvious fix is to go smaller. Figure 3 cuts one sentence four ways.

A table showing one sentence cut into words (6 units, open vocabulary), characters (39 units, over 130,000 possible Unicode symbols), UTF-8 bytes (39 units, 256 symbols) and cl100k_base subwords (8 units, 100,277 vocabulary)

Figure 3. "Safeguard scans software supply chains." at four granularities. The subword row is real cl100k_base output: the uncommon word "Safeguard" is spelled from three pieces while "software" and "supply" are single tokens.

Characters solve the unknown-word problem, but there are a lot of them. The GPT-2 paper notes that covering every Unicode symbol would need "a base vocabulary of over 130,000" before adding any multi-character tokens. Bytes cut the base set to 256, but a sentence becomes one symbol per byte, and every Devanagari or Japanese character takes three bytes in UTF-8. Since self-attention compares every position with every other position (post 1 covers why its cost grows with the square of the sequence length), long sequences are expensive.

Subwords are the compromise. You pick a vocabulary size, say 32,000 or 100,000, and a training algorithm decides which character sequences earn a slot. Frequent words get their own token. Rare words fall apart into frequent fragments. Anything at all can still be encoded, because the single characters or bytes are always in the vocabulary as a last resort. The three algorithms that dominate, BPE, WordPiece and the unigram language model, differ only in how they choose the slots and how they cut new text.

Where did these ideas come from?

The pieces arrived over three decades. Byte pair encoding began as a 1994 compression trick, WordPiece came from Google speech research in 2012, word2vec showed in 2013 what learned vectors can hold, and Sennrich et al. brought BPE to neural translation in 2016. The transformer arrived in 2017, then GPT-2's byte-level BPE in 2019 and RoPE and ALiBi in 2021.

Timeline from 1994 to 2024: Gage BPE compression 1994, WordPiece 2012, word2vec 2013, BPE for NMT and GNMT 2016, sinusoidal positions 2017, BERT WordPiece, unigram LM, SentencePiece and learned positions 2018, GPT-2 byte-level BPE 2019, RoPE and ALiBi 2021, LLaMA 2023, Llama 3 2024

Figure 2. The papers in this post, in order. Tokenizer work sits above the line, embeddings and positional encoding below.

The order explains a lot about modern models. Tokenization and positional encoding were both designed for machine translation systems with a few hundred million parameters. They were carried into LLMs a thousand times larger mostly because they worked well enough, and the later papers (byte-level BPE, RoPE, ALiBi) are patches for problems that only showed up at scale.

How does byte pair encoding (BPE) work?

BPE starts with every word split into characters, counts every adjacent pair of symbols across the training text, and merges the most frequent pair into a new symbol. It repeats that a fixed number of times. The ordered list of merges is the tokenizer: to encode new text, you replay the same merges in the same order.

The paper. Rico Sennrich, Barry Haddow and Alexandra Birch, University of Edinburgh, "Neural Machine Translation of Rare Words with Subword Units", posted to arXiv in August 2015 and published at ACL 2016 (arXiv 1508.07909).

The problem. Fixed-vocabulary translation models could not translate or produce words they had not seen, and the dictionary back-off workarounds assumed a one-to-one word mapping that often does not exist between languages.

The key idea. Borrow a 1994 compression algorithm by Philip Gage, which "iteratively replaces the most frequent pair of bytes in a sequence with a single, unused byte", and apply it to characters instead of bytes. Each merge creates a new symbol that stands for a character sequence, and frequent words end up as single symbols.

The details, as the paper states them:

  • Start from the character vocabulary. Write each word as characters plus a special end-of-word symbol, so the original spacing can be restored after translation.
  • Count all adjacent symbol pairs, weighted by word frequency, and replace every occurrence of the most frequent pair with a merged symbol.
  • Never merge across a word boundary. That lets the algorithm run on a word-frequency dictionary rather than the full text, which is much faster.
  • The final vocabulary size equals the number of starting characters plus the number of merges. The number of merges is the algorithm's only hyperparameter.

A worked BPE example

The paper's Algorithm 1 is a short Python listing that runs on a four-word toy corpus: "low" 5 times, "lower" 2 times, "newest" 6 times and "widest" 3 times. Here is the same logic in plain pseudocode, with the end-of-word marker written the way the paper's code writes it.

corpus = { "l o w </w>": 5, "l o w e r </w>": 2,
           "n e w e s t </w>": 6, "w i d e s t </w>": 3 }
merges = []
repeat NUM_MERGES times:
    counts = for every word, for every adjacent pair (a, b):
                 counts[(a, b)] += frequency of that word
    best = the pair with the highest count
    replace every "a b" in the corpus with "ab"
    merges.append(best)

Count the pairs by hand for the first step. The pair (e, s) appears in "newest" (6 times) and "widest" (3 times), so it scores 9. So do (s, t) and (t, end-of-word), for the same reason. The pair (l, o) scores 5 + 2 = 7. With a three-way tie at 9, the reference code takes the pair it counted first, which is (e, s). Running the paper's listing unchanged for ten merges gives exactly the sequence in Figure 4.

Table of ten BPE merges on the toy corpus: e+s, es+t, est+end, l+o, lo+w, n+e, ne+w, new+est end, low+end, w+i, with counts 9 9 9 7 7 6 6 6 5 3 and the corpus after each merge, plus the unseen word lowest encoded as low est

Figure 4. Ten BPE merges on the toy corpus from Sennrich et al. (2016), reproduced by running the paper's Algorithm 1. After ten merges "newest" is a single token, and the unseen word "lowest" encodes as "low" + "est" followed by the end-of-word marker.

Three things are worth seeing in that table.

First, the algorithm discovered the suffix "est" (with the end-of-word marker attached) before it discovered "low", because "est" occurs in two different words. BPE has no idea what a morpheme is. It finds them anyway, because morphemes are frequent.

Second, encoding a word nobody trained on still works. "lowest" never appeared, yet replaying the merges gives "low" plus "est" with the end-of-word marker attached, two tokens. The paper's own example is "lower" segmenting into "low" plus "er" when the merges come from a dictionary containing "low", "lowest", "newer" and "wider".

Third, symbols can become unreachable. A footnote in the paper notes the only test-time unknowns are unseen characters or "symbols of which all occurrences in the training text have been merged into larger symbols". Their example, by coincidence, is the fragment "safeguar", which in their training text always merged into "safeguard". That same effect, a vocabulary entry the model almost never sees during training, is what produces the glitch tokens covered later.

What the paper found

The experiments used WMT 2015 English to German (4.2 million sentence pairs, about 100 million tokens) and English to Russian (2.6 million sentence pairs, about 50 million tokens). The networks were attention-based recurrent models with a hidden layer of 1,000 units and a 620-dimensional embedding layer, trained with Adadelta on minibatches of 80 for about seven days, decoded with a beam of 12. The BPE vocabularies used 59,500 merges for separate source and target encodings and 89,500 merges for "joint BPE", where one set of merges is learned on both languages together so that names segment the same way on both sides. For Russian, the authors transliterated Cyrillic into Latin script to learn joint merges, then mapped them back.

The headline result: subword models beat a back-off dictionary baseline "by up to 1.1 and 1.3 BLEU" on English to German and English to Russian, and they translated rare and unseen words noticeably better.

Why it mattered, and what changed later

BPE made the open-vocabulary problem go away with a few dozen lines of code, and it had no linguistic rules at all. GPT-1 (2018) used a BPE vocabulary with 40,000 merges. GPT-2 changed the base alphabet from characters to bytes, covered below. LLaMA (2023) used BPE as implemented in SentencePiece. Almost every LLM you have used runs some version of this 2015 idea.

How is WordPiece different from BPE?

WordPiece builds its vocabulary bottom up like BPE, but it picks the next merge by how much it improves the likelihood of the training data rather than by raw pair count. At inference time BERT's version does not replay merges: it scans each word greedily, taking the longest vocabulary entry that matches, and marks word-internal pieces with "##".

WordPiece comes from Mike Schuster and Kaisuke Nakajima's "Japanese and Korean Voice Search" (ICASSP 2012), built for speech recognition in languages that do not separate words with spaces. Google described it in more detail in its 2016 neural machine translation paper, Wu et al. (arXiv 1609.08144), which made it famous.

As the GNMT paper describes it, the wordpiece model is trained "to maximize the language-model likelihood of the training data". Given a target count D, it selects D wordpieces so that the segmented corpus is as short as possible under the model, using a greedy algorithm. GNMT's version marks the start of each word with an underscore, so "Jet makers feud" becomes _J et _makers _fe ud. It limits the base character set to "roughly 500 for Western languages" and maps the rest to an unknown character. The authors found that "a total vocabulary of between 8k and 32k wordpieces" gave good BLEU and fast decoding across their language pairs, and their WPM-32K model, 32,000 wordpieces shared between source and target, performed well.

BERT (Devlin et al., 2018, arXiv 1810.04805) uses "WordPiece embeddings ... with a 30,000 token vocabulary". BERT flips GNMT's convention and marks the pieces that continue a word with "##" instead of marking word starts, so "playing" becomes play ##ing in the paper's own figure.

The BERT tokenizer's encoding step is simple enough to describe completely. Google's open source tokenization.py says it "uses a greedy longest-match-first algorithm". For each word, find the longest prefix that is in the vocabulary, emit it, then repeat on the remainder with "##" prepended, and give up with [UNK] if nothing matches. The file's own docstring example:

input:  "unaffable"
output: ["un", "##aff", "##able"]

There is a gap in the public record worth knowing about. Google never released WordPiece's training code. Hugging Face's course states plainly that "Google never open-sourced its implementation of the training algorithm of WordPiece", and describes its own best guess. In that version, each candidate pair gets this score:

score(a, b) = freq(ab) / ( freq(a) * freq(b) )

Dividing by the frequencies of the parts favours pairs whose parts are rare on their own but often appear together. Plain BPE would happily merge two very common symbols that just happen to sit next to each other a lot. So when someone says "WordPiece training works like X", they are usually describing the Hugging Face reconstruction.

What are SentencePiece and the unigram language model?

The unigram language model is a tokenizer algorithm by Taku Kudo (2018) that starts with a huge candidate vocabulary and prunes it, scoring each segmentation as a product of piece probabilities. SentencePiece, by Kudo and Richardson (2018), is a library that trains BPE or unigram tokenizers straight from raw text, treating spaces as ordinary symbols so encoding is lossless.

The unigram language model (Kudo, ACL 2018)

The paper. Taku Kudo, Google, "Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates", ACL 2018 (arXiv 1804.10959).

The problem. BPE is "based on a greedy and deterministic symbol replacement". It always produces one segmentation, but a sentence can be segmented many ways, and Kudo wanted to train models on several of them as a form of data augmentation.

The key idea. Assume each subword occurs independently. Then the probability of a segmentation is just the product of the probabilities of its pieces:

x = (x1, x2, ..., xM)        a segmentation into M pieces
P(x) = p(x1) * p(x2) * ... * p(xM)
best segmentation  x* = the x with the largest P(x)

Here p(xi) is the probability of piece xi in the vocabulary, and all the p values add up to 1. Locating the best segmentation sounds like it requires listing every possibility, but the Viterbi algorithm (dynamic programming) finds it in time proportional to the length of the text.

Figure 6 does this for a toy vocabulary. The word "unhappy" can be cut 24 ways using those pieces. "un | happy" scores 0.06 x 0.02 = 0.0012, a hundred times more likely than the runner-up "u | n | happy". The model prefers few, frequent pieces, without any rule telling it to.

A toy unigram vocabulary with probabilities and the six most probable of 24 segmentations of unhappy, led by un plus happy at 0.0012, plus the five-step training loop: seed, EM, loss per piece, keep top 80 percent, repeat

Figure 6. Unigram segmentation with illustrative probabilities. The bottom panel is the training loop from Kudo (2018).

Training works in the opposite direction to BPE. Instead of growing a vocabulary, you shrink one:

  1. Make "a reasonably big seed vocabulary": all characters plus the most frequent substrings in the corpus. The paper enumerates these with an Enhanced Suffix Array algorithm.
  2. With the vocabulary fixed, estimate every p(x) with the EM algorithm, which maximises the likelihood of the training text when the segmentation of each sentence is unknown.
  3. For each piece, compute loss_i, how much the total likelihood would fall if that piece were removed.
  4. Sort by loss and keep the top η percent ("η is 80, for example"), always keeping single characters so nothing becomes unencodable.
  5. Repeat steps 2 to 4 until the vocabulary is the size you want.

Subword regularization. Because the model assigns a probability to every segmentation, you can sample from them. During training, Kudo samples one segmentation per sentence per update from the top l candidates, with probabilities raised to a power α and renormalised. A small α samples almost uniformly. A large α almost always returns the Viterbi best. The neural model then sees "un happy", "u n happy" and "un hap py" for the same word across epochs and copes better with segmentation noise. Kudo reports consistent BLEU gains, especially on low-resource and out-of-domain settings.

Kudo also frames the difference neatly: BPE is a dictionary encoder that minimises the number of symbols, while the unigram model is an entropy encoder that minimises total code length, where the optimal code length of a symbol s with probability ps is -log ps (Shannon). That -log P column is what Figure 6 shows.

SentencePiece (Kudo and Richardson, EMNLP 2018)

The paper. Taku Kudo and John Richardson, Google, "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing", EMNLP 2018 system demonstrations (arXiv 1808.06226).

The problem. Earlier subword tools assumed the input was already split into words by a language-specific pre-tokenizer such as the Moses scripts. That works for English and badly for Chinese, Japanese or Korean, which do not put spaces between words, and it makes multilingual models fiddly.

The key ideas.

  • Train from raw sentences. No pre-tokenization, no language rules. It implements both BPE and the unigram model.
  • Lossless tokenization. SentencePiece treats input "just as a sequence of Unicode characters", including whitespace. It replaces each space with the visible meta symbol "▁" (U+2581), so "Hello world." can become [Hello] [▁wor] [ld] [.], and decoding is just "join the pieces and turn ▁ back into spaces". The paper states the guarantee as:
Decode(Encode(Normalize(text))) = Normalize(text)

By contrast, the older subword-nmt tool marks word-internal boundaries with "@@" and cannot represent runs of consecutive spaces, so it is not always reversible.

  • Normalization built in. Text is normalized with Unicode NFKC by default, with custom rules allowed, so "the same" string in different Unicode forms maps to the same tokens.
  • Self-contained models. The normalization rules, vocabulary and parameters live in one model file, so the exact same tokenization can be reproduced anywhere. That is good for reproducibility, and it is also why a tokenizer file is an artifact you should track like any other dependency.

The results. On the Kyoto Free Translation Task (Wikipedia articles, 440,000 training sentences), a shared 8,000-piece SentencePiece unigram model raised BLEU over an 80,000-word baseline from 28.24 to 29.55 for Japanese to English and from 20.06 to 21.62 for English to Japanese, with no external pre-tokenizer. On raw Japanese, segmentation ran about 380 times faster than subword-nmt.

Figure 5 puts the three algorithms side by side.

Comparison table of BPE, WordPiece and Unigram LM covering how training builds the vocabulary, how the next unit is chosen, how new text is segmented, the boundary marker, whether several segmentations are possible, and known users

Figure 5. BPE, WordPiece and the unigram model share a goal and differ in method. SentencePiece is a library that can train either BPE or unigram models; unigram is its default.

What is byte-level BPE, and why did GPT-2 adopt it?

Byte-level BPE runs the BPE algorithm on UTF-8 bytes instead of Unicode characters. The base vocabulary is just 256 byte values, so any string in any language, emoji or binary junk can be encoded with no unknown token. GPT-2 introduced it in 2019 with a 50,257-token vocabulary and a rule stopping merges across letters, digits and punctuation.

The paper. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever, OpenAI, "Language Models are Unsupervised Multitask Learners" (2019). It was published as an OpenAI report rather than on arXiv (PDF).

The problem. OpenAI wanted one model that could assign a probability to any string, so it could be evaluated on any dataset regardless of preprocessing. Character-level BPE needs every Unicode character in its base vocabulary to do that, which would mean a base of "over 130,000" symbols against the "32,000 to 64,000 token vocabularies often used with BPE". Pure byte-level language models existed but were "not competitive" with word-level ones on large datasets.

The key idea. Run BPE on bytes, so the base vocabulary has 256 entries. Then fix the one problem that causes. Applied naively, frequency-based merging wastes slots: the authors saw BPE learn many versions of common words "like dog since they occur in many variations such as dog. dog! dog?". So GPT-2 prevents BPE "from merging across character categories for any byte sequence", with "an exception for spaces". In practice the text is first split with a regular expression into runs of letters, runs of digits, runs of punctuation and contractions, each optionally carrying one leading space, and merges happen only inside those runs. The pattern from the original GPT-2 release, as preserved in tiktoken's source:

's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+

Read it piece by piece: common English contractions; an optional space followed by Unicode letters; an optional space followed by digits; an optional space followed by anything that is not a space, letter or digit; and whitespace. This is why " cat" with its leading space is one token in Figure 1. Attaching the space to the following word is cheap and keeps most words to a single token.

The architecture details. The GPT-2 vocabulary has 50,257 entries: 256 byte tokens, 50,000 merges and one special end-of-text token with ID 50256.

<|endoftext|>   ->   token ID 50256 in GPT-2's vocabulary

The context grew from GPT-1's 512 tokens to 1,024, and the largest model had 48 layers with a width of 1,600 (1.5 billion parameters). GPT-3 kept the same 50,257-token vocabulary (arXiv 2005.14165) with a 2,048-token context.

Why it mattered. Byte-level BPE removed the unknown token for good. The cost shows up in non-Latin scripts. A Devanagari letter is three bytes in UTF-8. If the training data held little Hindi, few merges cover those bytes, and a single letter can cost two or three tokens. Figure 7 measures that.

Grouped bar chart of token counts: English sentence 9, 8 and 8 tokens; Hindi greeting 32, 20 and 9; Japanese hello world 10, 4 and 2, for r50k_base, cl100k_base and o200k_base respectively

Figure 7. The same three texts in three generations of OpenAI tokenizers, measured with tiktoken 0.14.0. The English text is "Safeguard scans software supply chains.", the Hindi text is "नमस्ते, आप कैसे हैं?" and the Japanese text is "こんにちは世界".

The Hindi greeting is 50 bytes. The GPT-2 tokenizer needs 32 tokens for it, mostly two tokens per letter: one token for the shared first two bytes and one for the last byte. The GPT-4o tokenizer, with four times the vocabulary, needs 9.

What is tiktoken, and how do OpenAI's encodings differ?

tiktoken is OpenAI's open source byte-level BPE tokenizer library, written in Rust with Python bindings. It ships the exact vocabularies OpenAI's models use: r50k_base (GPT-2 and GPT-3, 50,257 tokens), p50k_base, cl100k_base (GPT-4 and GPT-3.5 Turbo, 100,277) and o200k_base (GPT-4o and later, 200,019). Its README claims it is 3 to 6 times faster than a comparable tokenizer.

The README describes BPE in three properties that are worth memorising: it is "reversible and lossless", it "compresses the text", and "on average, in practice, each token corresponds to about 4 bytes". For English, that last rule of thumb is a decent estimate of how many tokens a document will cost. The speed claim is measured on 1 GB of text against GPT2TokenizerFast from Hugging Face tokenizers 0.13.2.

Using it takes three lines:

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
print(enc.encode("The cat sat."))          # [791, 8415, 7731, 13]
print(enc.decode([791, 8415, 7731, 13]))   # The cat sat.

The encodings differ in more than size. The pre-tokenization regex changed between generations. In cl100k_base, the pattern matches digits one to three at a time, so a long number is chopped into chunks of at most three digits from the left, and leading whitespace is handled differently so that runs of spaces in code compress better. You can see the effect in Figure 8: "12345678" is 123 | 45 | 678 in the GPT-2 encoding and 123 | 456 | 78 in the GPT-4 encoding. In tiktoken 0.14.0's model table, gpt-4 and gpt-3.5-turbo map to cl100k_base, while gpt-4o, gpt-4.1, o1, o3 and gpt-5 map to o200k_base.

The n_vocab figures include special tokens. cl100k_base, for example, reserves IDs for end-of-text and for fill-in-the-middle markers used in code completion, which is why the totals are not round numbers.

There is a small supply chain detail in tiktoken's loader that most users never notice. The vocabulary files are downloaded on first use, and the library's encoding definitions pin each file's SHA-256 hash. If a downloaded or cached file does not match, it raises an error rather than silently using it. A tokenizer file decides which integers the model receives, so pinning it the same way you pin a package is the right instinct.

Other model families made the same move toward larger, tiktoken-style vocabularies. Meta's Llama 3 paper (arXiv 2407.21783) describes a 128,000-token vocabulary that "combines 100K tokens from the tiktoken tokenizer with 28K additional tokens to better support non-English languages", and reports that compression on a sample of English text improved "from 3.17 to 3.94 characters per token" compared with the 32,000-token Llama 2 tokenizer. More characters per token means the model "reads" more text for the same training compute and fits more text in the same context window.

How big should a tokenizer's vocabulary be?

There is no single right size. A bigger vocabulary shortens sequences, which saves attention compute and fits more text in the context, and it serves more languages fairly. It also adds parameters to the embedding and output layers and creates more rare tokens that the model barely sees during training. Published choices range from 8,000 to over 250,000.

Four forces pull against each other:

  • Sequence length. Every extra merge makes some text shorter. Llama 3's jump from 32,000 to 128,000 entries moved English from 3.17 to 3.94 characters per token, so the same 8,000-token window holds about a quarter more text.
  • Parameters. The embedding matrix has one row per token. Its size is V x d_model, where V is the vocabulary size. For Llama 3 8B that is 128,000 x 4,096, about 524 million numbers, or roughly 6.5 percent of the model. Figure 9 has more examples. If the output projection is a separate matrix, double it.
  • Softmax cost. At every generation step the model scores all V tokens. Bigger V, bigger final matrix multiply.
  • Under-trained tokens. Each token needs enough training examples for its embedding to be learned. Land and Bartolo (EMNLP 2024) found that, across the open models they tested, typically around 0.1 to 1 percent of the vocabulary consisted of severely under-trained tokens, with a large vocabulary and a tokenizer trained on different data from the model as the main risk factors.

Language coverage is the other big input. BigScience's BLOOM, built for 46 natural languages, tested 150,000 and 250,000 entries and settled on 250,680 to reach its "fertility" (tokens per word) target against monolingual tokenizers, noting that vocabulary size also fixes the embedding matrix size.

For a sense of history: GNMT's sweet spot was 8,000 to 32,000 wordpieces, BERT used 30,000, GPT-1 used 40,000 merges, GPT-2 and GPT-3 used 50,257, LLaMA used 32,000, and current frontier tokenizers sit between 100,000 and 256,000.

Why do LLMs struggle with numbers, other languages and odd strings?

Because the model never sees characters or digits, only token IDs. A number is cut into arbitrary chunks, so place value is never explicit. A word with or without a leading space or capital letter is a different, unrelated ID. Low-resource languages cost several times more tokens. And some vocabulary entries were barely seen in training, so they trigger erratic output.

Four panels with real token IDs: 12345678 split as 123 45 678 in r50k_base and 123 456 78 in cl100k_base versus LLaMA's single digits; token, space token and Token as IDs 5963, 4037 and 3404; space SolidGoldMagikarp as one token ID 43453 in r50k_base and five tokens in cl100k_base; and an arithmetic string split differently by each tokenizer

Figure 8. Four quirks, all visible in the token IDs. Every ID is real tiktoken output.

Numbers. "12345678" is three tokens in both OpenAI encodings, but different three tokens. Neither split lines up with thousands, hundreds or units. LLaMA's designers took the opposite approach. Their paper says plainly that they "split all numbers into individual digits, and fallback to bytes to decompose unknown UTF-8 characters". Singh and Strouse (2024, arXiv 2402.14903) compared the two styles on GPT-3.5 and GPT-4. Forcing right-to-left digit grouping, by inserting commas into numbers at inference time, "leads to largely improved performance" on arithmetic, and the errors under left-to-right grouping followed a systematic pattern. The gap shrank as models got larger.

Spelling and letter counting. Ask how many r's are in a word and the model has to recall the spelling of tokens it never sees as letters. " strawberry" might be one or several tokens, but none of them is a list of characters. The model can learn spellings from data, so it often gets these right, but the information is indirect.

Spaces and capitals. In cl100k_base, "token" is ID 5963, " token" is 4037 and "Token" is 3404. To the model these are three separate rows in the embedding matrix. Their vectors end up close because they appear in similar contexts, not because the model knows they share letters. Text with unusual spacing or casing lands on rarer tokens, which can change behaviour in small ways.

Other languages. Petrov, La Malfa, Torr and Bibi (NeurIPS 2023, arXiv 2305.15425) measured the same text translated into many languages. The tokenizer used by ChatGPT and GPT-4 "uses about 1.6 times more tokens to encode the same text in Italian as it does in English, 2.6 times for Bulgarian and 3 times for Arabic", and for Shan the gap "can be as high as 15 times". Even pure byte-level encoding makes some scripts over four times longer than others. More tokens means higher API cost, more latency and less room in the context window for the same content. Larger multilingual vocabularies, like o200k_base in Figure 7, narrow the gap.

Glitch tokens. In February 2023, Jessica Rumbelow and Matthew Watkins reported that GPT-3 era models behaved bizarrely when asked to repeat certain strings, the most famous being " SolidGoldMagikarp" (LessWrong). The cause is the train/tokenizer split from the start of this post. The GPT-2 tokenizer was trained on data where that Reddit username appeared often enough to earn its own token (ID 43453, as Figure 8 shows), but the string was rare in the language model's training data, so its embedding was barely updated.

Land and Bartolo's "Fishing for Magikarp" (EMNLP 2024, arXiv 2405.05417) turned this into a method. They combine tokenizer analysis, indicators computed from the model's embedding weights, and prompting tests to find under-trained tokens automatically, and they found them in every model they tested. Their recommendations are practical: train the tokenizer on data representative of the model's training data, encode and decode the whole vocabulary to find "unreachable" tokens that can never be produced from text, and check for under-trained tokens after small test runs. They also note that these tokens have "the potential to bypass guardrails", which matters for the security section below.

Is tokenization really to blame for LLM failures?

Partly. Tokenization explains specific, reproducible quirks: digit chunking, spelling, uneven cost across languages and glitch tokens. It does not explain most reasoning errors, and larger models increasingly work around it. Singh and Strouse found the arithmetic gap between tokenization styles shrinks with scale. Treat tokenization as one cause to test for, not a universal excuse.

The counter-argument deserves a fair hearing. "Blame the tokenizer" has become a reflex, and some of the failures attributed to it vanish when the same model is prompted to spell a word out first or when a larger model in the same family is used. Character-level and byte-level models avoid tokenization quirks entirely and still make mistakes. The honest version is narrower: tokenization changes what information is easy for the model to access, so it raises or lowers the difficulty of tasks that depend on characters, digits or rare strings. When you see a failure on exactly those inputs, re-encode the input (add spaces between digits, spell a word out, normalize Unicode) and test again before drawing conclusions.

What is an embedding matrix?

An embedding matrix is a learned table with one row per token in the vocabulary and one column per model dimension. Its shape is V x d_model. Looking up token ID i returns row i, a vector of d_model numbers that the model uses as that token's starting representation. The values start random and are learned during training like any other weight.

Figure 9 shows a toy matrix with 6 tokens and 4 dimensions. Token ID 3 selects row 3, the vector [0.9, -0.4, 0.1, 0.7]. That is the entire operation.

An illustrative 6 by 4 embedding matrix where token ID 3 selects row 3, shown as equivalent to multiplying a one-hot vector by the matrix, plus a table of real embedding sizes: BERT base 23.0M, GPT-2 80.4M, GPT-3 617.6M, Llama 3 8B 524.3M and Llama 3 405B 2.10B values

Figure 9. An embedding lookup selects one row. The table multiplies each paper's stated vocabulary size by its stated model width.

Mathematically, a lookup is the same as multiplying a "one-hot" vector by the matrix. A one-hot vector for ID 3 in a vocabulary of 6 has a 1 in position 3 and zeros everywhere else:

one_hot(3) = [0, 0, 0, 1, 0, 0]
one_hot(3) x E = row 3 of E = [0.9, -0.4, 0.1, 0.7]

Frameworks implement it as an indexed read, because multiplying by a vector of mostly zeros is a waste. The one-hot view is still useful, because it shows why the embedding layer is "just" a linear layer and why gradients only flow into the rows of tokens that appeared in the batch. That second fact is the mechanical reason under-trained tokens exist: a token that never appears in training data never receives an update, so its row stays near its random starting point.

Weight tying. The final layer of a language model turns each output vector back into a score for every token, which is a matrix of the same shape, V x d_model. Press and Wolf (EACL 2017, arXiv 1608.05859) showed that this "output embedding" is itself a valid word embedding and recommended tying it to the input embedding, using one matrix for both. The original transformer does exactly that: "we share the same weight matrix between the two embedding layers and the pre-softmax linear transformation", and it multiplies the embedding weights by the square root of d_model. Tying halves the vocabulary-related parameters. Many large models today keep the two matrices separate, trading parameters for flexibility.

Scale. For BERT base, 30,000 x 768 is about 23 million values, roughly a fifth of its 110 million parameters. For GPT-3 175B, 50,257 x 12,288 is about 618 million, a rounding error at 0.35 percent. For Llama 3 405B, 128,000 x 16,384 is about 2.1 billion. In small models the vocabulary choice is a large share of the parameter budget. In large ones it hardly matters.

What do embeddings actually capture?

Embeddings capture how a token is used. Training pushes tokens that appear in similar contexts toward similar vectors, so synonyms, related names and grammatical variants cluster together, and some relationships show up as consistent directions. In an LLM, the input embedding is only a context-free starting point. Meaning that depends on the sentence is built later, by the attention layers.

The clearest early evidence came from word2vec.

The papers. Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean, Google, "Efficient Estimation of Word Representations in Vector Space" (January 2013, arXiv 1301.3781), and Mikolov, Sutskever, Chen, Corrado and Dean, "Distributed Representations of Words and Phrases and their Compositionality" (NeurIPS 2013, arXiv 1310.4546).

The key idea. Drop the expensive hidden layers of earlier neural language models and train a shallow model whose only job is to produce good word vectors. Two architectures:

  • CBOW (continuous bag of words) predicts a word from the words around it.
  • Skip-gram predicts the surrounding words from the word in the middle.

The first paper reports that it "takes less than a day to learn high quality word vectors from a 1.6 billion words data set". The second paper added negative sampling (train the model to tell real context words from a handful of random ones rather than computing a full softmax over the vocabulary) and subsampling of very frequent words, which sped training up and improved rare-word vectors. It also learned vectors for phrases like "Air Canada", whose meaning is not the sum of "Air" and "Canada".

The famous result. Simple vector arithmetic recovered relationships:

vector("King") - vector("Man") + vector("Woman")   is closest to   vector("Queen")
vec("Madrid") - vec("Spain") + vec("France")        is closest to   vec("Paris")

Nobody told the model about gender or capitals. The directions emerged because "king" and "queen" appear in similar contexts that differ the way "man" and "woman" differ. The analogy test set in the first paper uses exactly these kinds of semantic and syntactic pairs (city to country, verb to past tense).

How that carries into LLMs. An LLM's input embedding table is trained the same basic way, by gradient descent on next-token prediction, so similar tokens end up nearby. But the vector for " bank" is identical whether the sentence is about rivers or money. The attention layers mix in context, and by the middle of the network the representation of " bank" in the two sentences has diverged. Researchers call these "contextual" representations, in contrast to the "static" ones from word2vec.

One point of vocabulary confusion is worth clearing up. The "embeddings" you store in a vector database for retrieval-augmented generation are different objects. They come from a separate embedding model that reads a whole passage and outputs one vector for it, usually taken from the model's final layer. They are much closer to the end of the pipeline than to the token embedding table. Post 8 in this series covers retrieval in detail.

Why does a transformer need positional encoding?

Self-attention treats its input as a set. Every token compares itself with every other token using the same weights, so shuffling the tokens just shuffles the outputs. Without extra information, "dog bites man" and "man bites dog" look identical to the attention layers. Positional encoding injects each token's position so word order can influence the result.

Recurrent networks got order for free, because they read one token at a time. Convolutions got local order from their sliding windows. The transformer removed both. In the words of Vaswani et al., "since our model contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens in the sequence."

Here is why, concretely. An attention score between positions i and j is a dot product of a query vector from token i and a key vector from token j, both computed from the token's embedding with the same weight matrices at every position. Nothing in that computation knows what i or j are. Swap two tokens and their scores swap with them. Post 1 walks through the full attention computation.

There are two broad ways to fix this, and the four methods in the rest of this post split along that line. Absolute methods give every position its own signal, added to the token vector once at the bottom of the network (sinusoidal and learned). Relative methods make the attention score itself depend on the distance between two tokens, inside every layer (RoPE and ALiBi).

How does sinusoidal positional encoding work?

Sinusoidal encoding gives each position a fixed vector of sine and cosine values at many different frequencies and adds it to the token's embedding. Low dimensions oscillate quickly, high dimensions slowly, so every position gets a unique pattern. Nothing is learned. The transformer paper chose it in 2017 because a fixed offset between positions corresponds to a fixed linear transformation.

The paper. Vaswani et al., "Attention Is All You Need", NeurIPS 2017, section 3.5 (arXiv 1706.03762).

The formula. For a position pos and a dimension index:

PE(pos, 2i)   = sin( pos / 10000^(2i / d_model) )
PE(pos, 2i+1) = cos( pos / 10000^(2i / d_model) )

Each pair of dimensions (2i and 2i+1) is one "clock" running at its own speed. For pair 0, the angle is just pos, one radian per position, which repeats every 2π, about 6.3 positions. Each later pair runs slower by a constant factor. The paper describes it as wavelengths forming "a geometric progression from 2π to 10000 · 2π".

A worked example. Take a tiny model with d_model = 4, so there are two clocks. For pair i = 0 the divisor is 10000^0 = 1. For pair i = 1 it is 10000^(2/4) = 100.

pos = 0:  [sin 0,  cos 0,  sin 0,    cos 0   ] = [0.000,  1.000, 0.000, 1.000]
pos = 1:  [sin 1,  cos 1,  sin 0.01, cos 0.01] = [0.841,  0.540, 0.010, 1.000]
pos = 2:  [sin 2,  cos 2,  sin 0.02, cos 0.02] = [0.909, -0.416, 0.020, 1.000]
pos = 3:  [sin 3,  cos 3,  sin 0.03, cos 0.03] = [0.141, -0.990, 0.030, 1.000]

The first clock spins fast and distinguishes neighbours. The second barely moves over four positions but would separate position 10 from position 300. In a real model with d_model = 512, there are 256 clocks covering everything from a handful of tokens to tens of thousands. The vector is simply added to the token embedding, element by element, before the first layer.

Heatmap of sinusoidal positional encodings for positions 0 to 39 and dimensions 0 to 63, showing fast oscillation in low dimensions and slow change in high dimensions, with a worked d equals 4 example for position 1

Figure 10. The sinusoidal encoding for 40 positions at d = 64. Each row is one position's vector. The left columns flicker, the right columns barely change, and together they give every position a distinct fingerprint.

Why sines and cosines? The authors "hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset k, PE(pos+k) can be represented as a linear function of PE(pos)". Here is why that is true. Look at one clock, with angle a = pos x w for some frequency w. Moving forward k positions adds k x w to the angle. The angle-addition identities give:

sin(a + kw) =  cos(kw) * sin(a) + sin(kw) * cos(a)
cos(a + kw) = -sin(kw) * sin(a) + cos(kw) * cos(a)

The numbers cos(kw) and sin(kw) depend only on the offset k, not on the position. So shifting by k is the same 2 x 2 rotation applied to every clock pair at every position. Hold on to that idea, because RoPE builds directly on it.

What the paper found. The authors also tried learned positional embeddings and "found that the two versions produced nearly identical results". In their Table 3, the base model with sinusoids scored 4.92 perplexity and 25.8 BLEU on the English to German development set, and the learned variant (row E) scored 4.92 and 25.7. They kept sinusoids "because it may allow the model to extrapolate to sequence lengths longer than the ones encountered during training".

What later work found. That extrapolation hope did not hold up well. When Press, Smith and Lewis tested it four years later, sinusoidal models degraded quickly once inputs grew past the training length, which is part of what motivated ALiBi.

What are learned positional embeddings?

Learned positional embeddings replace the fixed sinusoids with a second trainable table, one row per position up to a maximum length. Row 0 is added to the first token, row 1 to the second, and so on. GPT-1, GPT-2 and BERT all used them. They work well within the trained length but cannot represent any position past the table's last row.

GPT-1 (Radford et al., 2018) states it directly: "We used learned position embeddings instead of the sinusoidal version proposed in the original work." Its model had 12 layers, 768-dimensional states and 512-token training sequences. BERT likewise adds "learned" position embeddings (plus a segment embedding that marks sentence A or B) to its token embeddings, with sequences of up to 512 tokens. GPT-2 grew the table to 1,024 positions and GPT-3 to 2,048.

The appeal is flexibility: the model can learn whatever position pattern helps, and the transformer paper's experiment showed no loss in quality. The cost is a hard ceiling. As the ALiBi paper puts it, "the learned positional embedding approach does not have a way to encode positions greater than L". Position 2,049 simply has no row in GPT-3's table. The table also adds max_len x d_model parameters, though that is small next to the token embedding matrix.

As models moved to longer contexts, both absolute schemes gave way to relative ones. The two that won are RoPE and ALiBi.

How does RoPE (rotary position embedding) work, step by step?

RoPE encodes position by rotating the query and key vectors inside every attention layer, by an angle proportional to the token's position. Because the attention score is a dot product, and rotating both vectors changes their dot product only by the difference of their angles, the score ends up depending on the distance between two tokens rather than their absolute positions. Nothing is learned.

The paper. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen and Yunfeng Liu, Zhuiyi Technology, "RoFormer: Enhanced Transformer with Rotary Position Embedding", first posted 20 April 2021 (arXiv 2104.09864).

The problem. Absolute encodings add position to the content vector, so the model has to untangle "what" from "where" before it can reason about distance. Relative encodings existed, but they modified the expanded attention formula in ways that were awkward to combine with efficient attention variants.

The key idea. Find functions that attach position m to a query and position n to a key such that their inner product depends only on the content and on the offset, m minus n. The paper writes the goal as:

< f_q(x_m, m), f_k(x_n, n) > = g(x_m, x_n, m - n)

Here x_m is the token vector at position m, f_q and f_k produce the position-aware query and key, the angle brackets mean dot product, and g is some function that sees only the offset. The answer turns out to be rotation.

Step 1: the two-dimensional case

Suppose a query or key has only two numbers, so it is an arrow in a flat plane. Pick a fixed angle θ. RoPE rotates the query for position m by m x θ, and the key for position n by n x θ. A 2-D rotation by angle a is this matrix multiply:

rotate(v, a) = [ cos a   -sin a ] [ v1 ]
               [ sin a    cos a ] [ v2 ]

The paper's equation 13 is exactly this, applied after the usual query or key projection W_q x_m or W_k x_n.

Step 2: why only the difference survives

The dot product of two arrows equals their lengths multiplied together times the cosine of the angle between them. Rotation does not change length. If you rotate the query by m x θ and the key by n x θ, the angle between them changes by (n minus m) times θ, regardless of what m and n are individually. So the score depends on the content and on the offset, and nothing else.

Step 3: a worked toy example

Use θ = 30 degrees, a query q = (1, 0) and a key k = (0.6, 0.8). Before rotation, their dot product is 1 x 0.6 + 0 x 0.8 = 0.6.

Case A: query at m = 3, key at n = 1
  q' = rotate(q, 90 deg)  = ( 0.000, 1.000)
  k' = rotate(k, 30 deg)  = ( 0.120, 0.993)
  q' . k' = 0.000 x 0.120 + 1.000 x 0.993 = 0.993

Case B: query at m = 5, key at n = 3
  q' = rotate(q, 150 deg) = (-0.866, 0.500)
  k' = rotate(k, 90 deg)  = (-0.800, 0.600)
  q' . k' = 0.693 + 0.300 = 0.993

Case C: query at m = 1, key at n = 3   (key is now ahead of the query)
  q' . k' = -0.393

Cases A and B use completely different rotated vectors, yet give the same score, because the offset is 2 in both. Case C has the same distance in the other direction and gives a different score, so RoPE also knows which token came first.

Two panels showing q equals 1 0 and k equals 0.6 0.8 rotated by 30 degrees per position: query at 3 and key at 1, then query at 5 and key at 3, both with dot product 0.993, plus a table of per-pair frequencies 1, 0.1, 0.01 and 0.001 for d equals 8

Figure 11. RoPE in two dimensions. The dashed arrows are the original q and k. Different absolute positions, same offset, same score.

Step 4: the general case with d dimensions

Real query and key vectors have dozens or hundreds of dimensions. RoPE splits a vector of width d into d/2 pairs and rotates each pair by its own angle, using a different speed per pair:

pair i  (i = 1, 2, ..., d/2)  is rotated by angle  m * theta_i
theta_i = 10000^( -2(i - 1) / d )

That is the same geometric ladder of frequencies as the sinusoidal encoding. For d = 8 the four speeds are 1, 0.1, 0.01 and 0.001 radians per position, with wavelengths of about 6.3, 63, 628 and 6,283 positions. The paper writes the whole thing as one block-diagonal matrix R(Θ, m) with a 2 x 2 rotation block for each pair and zeros elsewhere. Applied to attention, it gets:

q_m . k_n  =  ( R(m) W_q x_m ) . ( R(n) W_k x_n )
           =  x_m^T  W_q^T  R(n - m)  W_k  x_n

The middle term, a rotation by n minus m, contains only the offset. The rotation matrix is orthogonal, which the authors note "ensures stability during the process of encoding position information" because it never stretches or shrinks the vector.

Step 5: how it is implemented

Building a mostly-zero d x d matrix would waste time, so the paper gives an element-wise version (its equation 34). In plain pseudocode, for one vector x at position m:

for each pair index i from 1 to d/2:
    a = m * theta_i
    x1, x2 = x[2i - 1], x[2i]          # the two numbers in pair i
    out[2i - 1] = x1 * cos(a) - x2 * sin(a)
    out[2i]     = x1 * sin(a) + x2 * cos(a)

The cosines and sines depend only on position, so implementations precompute them once for every position and reuse them in every layer. RoPE is applied to queries and keys only, never to values, and never to the input embedding.

Step 6: the long-term decay property

The paper proves that, with these frequencies, the upper bound on the attention score between two tokens shrinks as the distance between them grows. Intuitively, at large offsets the fast-spinning pairs point in essentially random directions relative to each other and their contributions cancel. This gives the model a gentle built-in preference for nearby tokens, which matches how language works, without forbidding long-range attention.

What the paper found

On WMT 2014 English to German translation, RoFormer scored 27.5 BLEU against 27.3 for the Transformer base model. Replacing BERT's position encoding with RoPE during pre-training gave faster convergence of the masked language modelling loss. On a Chinese legal case-matching task with long documents (CAIL2019-SCM), RoFormer with a 1,024-token input reached 69.79 percent test accuracy versus 68.10 for WoBERT at 512 tokens. The authors were candid about the limits: they lacked "thorough explanations on why it converges faster" than other position schemes.

Why it mattered

RoPE spread through open models quickly. GPT-J popularized it in 2021, GPT-NeoX-20B adopted it "instead of the learned positional embeddings used in GPT models", and LLaMA lists "Rotary Embeddings" among its architecture changes. Llama 3 kept RoPE and raised the base from 10,000 to 500,000, which slows every clock down and helps with longer contexts. Methods that stretch RoPE to longer contexts after training, such as positional interpolation, are covered in post 9 of this series.

How does ALiBi work?

ALiBi (Attention with Linear Biases) adds no position vectors at all. Instead, inside each attention head it subtracts a penalty from every query-key score proportional to the distance between the two tokens, using a fixed slope per head. Nearby tokens are penalised a little, distant tokens a lot. The ALiBi paper showed it lets models run on longer inputs than they were trained on.

The paper. Ofir Press (University of Washington and Facebook AI Research), Noah A. Smith (University of Washington and Allen Institute for AI) and Mike Lewis (Facebook AI Research), "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation", ICLR 2022, first posted August 2021 (arXiv 2108.12409).

The problem. Training on long sequences is expensive because attention cost grows with the square of the length. It would be cheaper to train short and run long, but the authors showed that the then-standard position methods "do not allow for efficient extrapolation". Sinusoidal models degraded soon after the training length. Rotary and T5-style relative bias extrapolated somewhat better but trained and ran more slowly.

The key idea. For query position i, ALiBi computes the usual scores against keys 1 to i and adds a fixed, non-learned bias:

scores_i = q_i K^T  +  m * [ -(i - 1), ..., -2, -1, 0 ]
weights_i = softmax( scores_i )

m is a slope that is different for each head and fixed before training. The bias is zero for the token itself and drops by m for every step back. No position embeddings are added "at any point in the network".

The slopes. For a model with 8 heads, the slopes are the geometric sequence 1/2, 1/4, 1/8, and so on down to 1/256. For n heads in general, the sequence "starts at 2^(-8/n) and uses that same value as its ratio". With 16 heads that gives 1/2^0.5, 1/2^1, 1/2^1.5 and so on to 1/2^8. Steep heads attend locally and gentle heads see far back. The authors tried learnable slopes and found they "did not yield strong extrapolation results", and also slowed training by 3 percent.

A worked ALiBi example

Take query 5 attending to keys 1 to 5 with raw scores 2.0, 1.0, 0.5, 1.5 and 1.0. Without any bias, softmax turns those into attention weights of 0.390, 0.143, 0.087, 0.236 and 0.143. Most weight goes to key 1, the furthest token.

Now add the bias for a head with slope m = 1/2. The distances are 4, 3, 2, 1 and 0, so the biases are -2, -1.5, -1, -0.5 and 0:

raw scores:        2.0    1.0    0.5    1.5    1.0
bias (m = 1/2):   -2.0   -1.5   -1.0   -0.5    0.0
biased scores:     0.0   -0.5   -0.5    1.0    1.0
softmax weights:   0.131  0.079  0.079  0.355  0.355

Attention has shifted to the two most recent tokens. For the gentlest head, m = 1/256, the biases are tiny and the weights barely move (0.387, 0.143, 0.087, 0.238, 0.145), so that head can still look far back.

A 5 by 5 causal matrix of ALiBi biases from 0 to minus 4 times the slope, the eight head slopes from one half to one over 256, and a worked softmax table showing attention shifting to recent tokens for slope one half and barely changing for slope one over 256

Figure 12. ALiBi's bias matrix and a worked example. The softmax numbers are this figure's own computation; the bias rule and slopes are from the paper.

What the paper found

The development experiments used the WikiText-103 benchmark (about 103 million training tokens from English Wikipedia) with a 16-layer model of width 1,024, 8 heads and a feed-forward inner size of 4,096, with tied input and output embeddings. The headline experiment scaled up to a 1.3 billion parameter model with 25 layers, 16 heads, width 2,048 and feed-forward size 8,192, trained on 461 GB of text. That model, trained on 1,024-token inputs, extrapolated to 2,048 tokens and achieved "the same perplexity as a sinusoidal position embedding model trained on inputs of length 2048 but training 11% faster and using 11% less memory". The paper also attributes some of ALiBi's gains to its "inductive bias towards recency".

Why it mattered

ALiBi showed that a position method could be almost free: no parameters, no extra layers and no change to the values, just a constant matrix added before the softmax. BigScience's BLOOM adopted it, explaining that ALiBi "directly attenuates the attention scores based on how far away the keys and queries are". RoPE ended up more common in later open models, but ALiBi's framing of length extrapolation as a design goal shaped the long-context work covered in post 9.

Which positional encoding should you use?

For a new decoder-only LLM today, RoPE is the default in most open model families, usually with a raised base frequency and an interpolation method for long contexts. ALiBi is the simplest relative option and extrapolates well in its paper's tests. Learned and sinusoidal absolute encodings remain fine for encoders and short, fixed-length tasks.

Comparison table of sinusoidal, learned, RoPE and ALiBi positional encodings: where the signal enters, trained parameters, absolute or relative, behaviour beyond training length, and example models

Figure 13. The four methods side by side. "Longer than training?" summarizes what each paper or the ALiBi comparison reported.

The trade-offs in Figure 13 come down to two questions. First, does position enter once at the bottom or in every layer? Once at the bottom is simplest but the signal must survive many layers of mixing. Every layer is sturdier, and it is what both modern methods do. Second, is the signal absolute or relative? Language mostly cares about relative order ("the adjective just before this noun"), and relative methods generalize more naturally to positions the model never trained on.

No single paper settled this, and the honest answer is that the right choice depends on your context-length needs and your inference stack. Most people never choose: they use whatever their base model was trained with, because changing positional encoding means retraining.

What security risks live in tokenizers and embeddings?

Several are documented. Invisible Unicode characters can carry instructions a human reviewer cannot see. Small spelling changes can alter how a guard classifier tokenizes a prompt and slip past it. Unusual tokenizations and glitch tokens can push models off their trained behaviour. Stored embeddings can be inverted back into text. And custom tokenizer code can run when a model loads.

This layer is easy to overlook in a security review because it looks like plumbing. Every item below is documented in a paper or a public write-up. None of the descriptions includes attack steps.

A table mapping seven documented risks to pipeline stages with sources and defences: invisible Unicode tag characters, TokenBreak, adversarial tokenization, glitch tokens, adversarial suffixes, embedding inversion, and custom tokenizer or model code

Figure 14. Documented risks at the tokenizer and embedding layers. The defences are general practice drawn from the cited sources, not a guarantee.

Invisible instructions in Unicode. The Unicode "Tags" block contains characters that mirror ASCII but are usually not rendered. In January 2024, Johann Rehberger described how text written in these characters is invisible to people yet still read by LLMs, crediting an earlier discovery by Riley Goodside, and demonstrated hidden instructions being acted on in ChatGPT (Embrace The Red). Because a byte-level tokenizer encodes every code point, nothing in the tokenizer stops it. His recommended mitigation is to filter these code points from prompts and responses. This is one concrete form of indirect prompt injection; our OWASP LLM01 prompt injection checklist covers the wider class.

Guard classifiers fooled by tokenization. Many teams screen prompts with a small text classifier before they reach the main model. TokenBreak (Schulz, Yeung and Evans, 2025, arXiv 2506.07948) showed that a tiny change, such as one prepended letter turning "Reveal password" into "aReveal password", can change how a BPE or WordPiece classifier tokenizes the text so that it misses a malicious input, while the larger model downstream still understands it. In their tests, classifiers using BPE and WordPiece were susceptible and those using Unigram tokenizers were not. Their proposed defence pre-tokenizes the input with a Unigram tokenizer and maps the result to the classifier's own tokens, without retraining.

Adversarial tokenization. A string has one canonical tokenization, but many others decode to the same text. Geh, Shao and Van den Broeck (ACL 2025, arXiv 2503.02174) found that models trained on canonical tokenizations "still retain semantic understanding of other tokenizations", and that feeding a harmful request in a non-canonical tokenization is "an effective yet previously neglected axis of attack", competitive with other methods "without changing the text of the harmful request". This matters wherever an application passes token IDs rather than text to a model.

Glitch and under-trained tokens. As covered above, Land and Bartolo note that under-trained tokens "can potentially be exploited to more easily circumvent guardrails" and recommend checking for them before deployment.

Optimized adversarial suffixes. Zou et al. (2023, arXiv 2307.15043) used "a combination of greedy and gradient-based search techniques" over tokens to find suffixes that push aligned models to comply with harmful requests. Suffixes found on open models transferred to the public interfaces of commercial chat assistants. The search operates directly in token space, which is why these strings often look like gibberish to people.

Embedding inversion. It is tempting to treat stored vectors as anonymized. Morris, Kuleshov, Shmatikov and Rush (EMNLP 2023, arXiv 2310.06816) showed a method that iteratively corrects and re-embeds text and recovers "92% of 32-token text inputs exactly" from their embeddings, including full names from clinical notes. If your vector database holds embeddings of sensitive documents, protect it like it holds the documents.

Tokenizer and model files that run code. A tokenizer is usually a data file, but Hugging Face's loaders can execute custom Python shipped in a model repository when trust_remote_code=True is set. The Transformers documentation advises extra caution with custom models and suggests loading "from a specific revision" (a commit hash) "to avoid loading model code that may have changed". The same pinning logic applies to tokenizer vocabularies, which is what tiktoken's built-in hash checks do.

A practical checklist for teams building on LLMs:

  • Normalize and filter input text: strip or flag Unicode tag characters and other invisible code points, at the prompt and in model output.
  • Do not rely on a single classifier to block prompt injection. Combine input screening with output checks and least-privilege tool permissions.
  • If your application sends token IDs to a model, re-encode them canonically from text.
  • Pin tokenizer and model revisions by hash, and review any custom code before enabling it.
  • Treat vector stores as sensitive data stores, with the same access controls as the source documents.
  • Test new models for under-trained tokens and odd tokenizations as part of evaluation.

How do you secure what you build on these models?

The model's internals are only part of the risk. Teams also pull model files from public hubs, depend on fast-moving AI SDKs, give agents tools and data access, and connect assistants to MCP servers. Each of those is a supply chain or runtime surface that can be inventoried, scanned and governed like any other software dependency.

This is where Safeguard fits. It is a short list, and each item links to a deeper post.

The next post in this series goes inside attention itself: scaled dot-product attention, multi-head attention, causal masking, multi-query and grouped-query attention, FlashAttention and the KV cache.

Try it yourself

Click through these interactive Safeguard demos to see the steps described in this guide.

Test yourself

How does byte pair encoding build a tokenizer?

It starts with words split into characters, counts every adjacent pair, and merges the most frequent pair into a new symbol. It repeats this a fixed number of times, and the ordered list of merges is the tokenizer.

Why did GPT-2 use byte-level BPE?

The base vocabulary is just 256 byte values, so any string in any language can be encoded with no unknown token. The cost shows up in non-Latin scripts, where a single letter can take two or three tokens.

Why do LLMs struggle with numbers and spelling?

The model sees token IDs, not digits or letters. A number is cut into arbitrary chunks, so place value is never explicit, and token, space token and Token are three separate rows in the embedding matrix.

What is an embedding matrix?

It is a learned table with one row per token and one column per model dimension, shaped V by d_model. Looking up token ID i returns row i, and a token that never appears in training never gets updated, which is how under-trained tokens arise.

How does RoPE encode position?

It rotates the query and key vectors by an angle proportional to the token's position. Because rotation changes the dot product only by the difference of the angles, the score depends on the distance between two tokens, and nothing is learned.

Name two documented security risks at the tokenizer and embedding layers.

Invisible Unicode tag characters can carry hidden instructions that humans cannot see. Stored embeddings can also be inverted, and one study recovered 92 percent of 32-token inputs exactly, so a vector database should be protected like the documents.

Frequently asked questions

What is a token in an LLM?

A token is one entry in the model's fixed vocabulary, usually a common word, a word fragment, a punctuation mark or a single byte. The tokenizer converts text into a list of token IDs, and the model reads and writes only those IDs. In OpenAI's tokenizers a token averages about 4 bytes of English text, according to the tiktoken README.

How many words is 1,000 tokens?

It depends on the tokenizer and the language. Using tiktoken's rule of thumb of about 4 bytes per token, 1,000 tokens is roughly 4,000 characters of ordinary English, which is typically around 700 to 800 words. Code, numbers, unusual words and non-Latin scripts usually take more tokens per word. Count with the model's own tokenizer when cost or context limits matter.

What is the difference between BPE, WordPiece and SentencePiece?

BPE and WordPiece are algorithms that grow a vocabulary by merging symbols, BPE by pair frequency and WordPiece by likelihood gain. The unigram model is a third algorithm that prunes a large vocabulary down. SentencePiece is a library, not an algorithm: it trains BPE or unigram tokenizers directly from raw text and treats spaces as ordinary symbols.

Why do LLMs struggle with spelling and counting letters?

Because they never see letters. A word arrives as one or a few token IDs, and the model has to have memorized how each token is spelled from patterns in its training data. It often has, but the information is indirect, so tasks such as counting letters or reversing a word are harder than they look. Asking the model to spell the word out first usually helps.

Are token embeddings the same as the embeddings used for RAG?

No. Token embeddings are rows of the model's input table, one fixed vector per token, used as the starting point inside the model. RAG embeddings come from a separate embedding model that reads a whole passage and produces one vector for it, which is stored in a vector database for search. Both are learned vectors, but they are made differently and used for different jobs.

Why does a transformer need positional encoding?

Self-attention compares every token with every other token using the same weights at every position, so it has no built-in notion of order. Shuffle the input and the outputs shuffle the same way. Positional encoding adds information about where each token sits, either as a vector added at the input (sinusoidal or learned) or inside attention itself (RoPE or ALiBi).

What is the difference between RoPE and ALiBi?

Both make attention depend on the distance between tokens and neither adds learned parameters. RoPE rotates query and key vectors by angles proportional to position, so the dot product reflects their offset. ALiBi leaves the vectors alone and subtracts a fixed penalty proportional to distance from each attention score, with a different slope per head. RoPE is more widely used in recent open models.

Can a tokenizer be a security risk?

Yes, in documented ways. Invisible Unicode characters can smuggle instructions past human reviewers, small spelling changes can alter how a guard classifier tokenizes a prompt, and under-trained glitch tokens can push models into odd behaviour. Tokenizer and model files can also bring custom code that runs at load time. Normalize input, pin files by hash and layer your defences.

Papers and sources

The papers this post explains, with links to the original arXiv pages or official PDFs:

  • Sennrich, Haddow and Birch (2016). Neural Machine Translation of Rare Words with Subword Units. ACL 2016. arXiv 1508.07909
  • Schuster and Nakajima (2012). Japanese and Korean Voice Search. ICASSP 2012. Google Research
  • Wu et al. (2016). Google's Neural Machine Translation System. arXiv 1609.08144
  • Kudo (2018). Subword Regularization. ACL 2018. arXiv 1804.10959
  • Kudo and Richardson (2018). SentencePiece. EMNLP 2018. arXiv 1808.06226
  • Devlin et al. (2018). BERT. arXiv 1810.04805
  • Radford et al. (2018). Improving Language Understanding by Generative Pre-Training (GPT-1). PDF
  • Radford et al. (2019). Language Models are Unsupervised Multitask Learners (GPT-2). PDF
  • Brown et al. (2020). Language Models are Few-Shot Learners (GPT-3). arXiv 2005.14165
  • OpenAI tiktoken. GitHub
  • Mikolov et al. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv 1301.3781
  • Mikolov et al. (2013). Distributed Representations of Words and Phrases and their Compositionality. NeurIPS 2013. arXiv 1310.4546
  • Press and Wolf (2017). Using the Output Embedding to Improve Language Models. EACL 2017. arXiv 1608.05859
  • Vaswani et al. (2017). Attention Is All You Need. NeurIPS 2017. arXiv 1706.03762
  • Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv 2104.09864
  • Press, Smith and Lewis (2021). Train Short, Test Long (ALiBi). ICLR 2022. arXiv 2108.12409
  • Black et al. (2022). GPT-NeoX-20B. arXiv 2204.06745
  • BigScience Workshop (2022). BLOOM. arXiv 2211.05100
  • Touvron et al. (2023). LLaMA. arXiv 2302.13971
  • Llama Team (2024). The Llama 3 Herd of Models. arXiv 2407.21783
  • Petrov et al. (2023). Language Model Tokenizers Introduce Unfairness Between Languages. NeurIPS 2023. arXiv 2305.15425
  • Singh and Strouse (2024). Tokenization counts. arXiv 2402.14903
  • Land and Bartolo (2024). Fishing for Magikarp. EMNLP 2024. arXiv 2405.05417
  • Zou et al. (2023). Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv 2307.15043
  • Morris et al. (2023). Text Embeddings Reveal (Almost) As Much As Text. EMNLP 2023. arXiv 2310.06816
  • Geh, Shao and Van den Broeck (2025). Adversarial Tokenization. ACL 2025. arXiv 2503.02174
  • Schulz, Yeung and Evans (2025). TokenBreak. arXiv 2506.07948

Token IDs and token counts in this post were produced with tiktoken 0.14.0. The BPE merge table was produced by running Algorithm 1 from Sennrich et al. unchanged. All other worked numbers are toy examples computed for this post and labelled as such.

Never miss an update

Weekly insights on software supply chain security, delivered to your inbox.

We email you a link to confirm, and only then add you to the list. Unsubscribe in one click. Privacy

Self-healing security runs on Safeguard.

Your first fix PR is minutes away.

No sales call required, even your agent can complete the purchase over MCP.