Attention is the operation that lets every token in a language model pull information from every earlier token, weighted by how relevant each one looks. It is a softmax over dot products between a query and a set of keys, used to average a set of values. Almost everything else in this post is about making that one operation cheaper: fewer bytes in the KV cache (multi-query and grouped-query attention), fewer trips to slow GPU memory (FlashAttention), less wasted cache memory on a serving box (PagedAttention), and fewer token pairs to compare at all (sliding window and sparse attention).
This is part 3 of our LLM architecture series. Part 1 walks through the original Transformer from "Attention Is All You Need". Part 2 covers how text becomes tokens, embeddings and positions. This part zooms in on the attention block itself. It is written for engineers who build on LLMs and want to understand what the model is actually computing, and why inference servers are designed the way they are. You need high-school algebra and a rough idea of what a matrix multiply is. Every symbol is defined when it first appears, and the arithmetic is shown.
By the end you should be able to compute a small attention pass by hand, explain why the scores are divided by the square root of the head size, work out how much GPU memory a model's KV cache needs for a given context length, and say in plain words what FlashAttention and PagedAttention change and what they leave alone.
Figure 1. The twelve papers this post covers, by first arXiv submission date. Purple papers define the mechanism, cyan papers make it cheaper to run, and amber papers reduce how many token pairs get compared.
Contents
- What problem did attention originally solve?
- How did Luong attention differ from Bahdanau attention?
- What is scaled dot-product attention?
- Why is attention scaled by the square root of d_k?
- How does causal masking work?
- What is multi-head attention and why use several heads?
- What do attention heads actually learn?
- Why is generating text slow even on a fast GPU?
- What is the KV cache and how much memory does it use?
- What is multi-query attention?
- What is grouped-query attention?
- How does FlashAttention speed up attention without approximating it?
- What did FlashAttention-2 change?
- What is PagedAttention and why does vLLM use it?
- What are sliding window and sparse attention?
- How do these techniques fit together in a modern LLM?
- What does this mean for securing what you build on these models?
- FAQ
What problem did attention originally solve?
Attention was introduced in 2014 to fix a bottleneck in neural machine translation: the encoder had to squeeze a whole source sentence into one fixed-length vector. Bahdanau, Cho and Bengio let the decoder look back at every source word for each word it produced, using learned weights, and long-sentence translation improved sharply.
The paper
"Neural Machine Translation by Jointly Learning to Align and Translate" by Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio was posted to arXiv in September 2014 (arXiv 1409.0473) and accepted at ICLR 2015 as an oral presentation.
The problem it solved
In 2014 the leading neural translation models were encoder-decoders built from recurrent neural networks (RNNs). An RNN reads a sentence one word at a time and carries a hidden state, a vector that is updated after each word. The encoder's final hidden state was handed to the decoder as the summary of the whole sentence, and the decoder generated the translation from it.
The authors argued that this fixed-length vector was the bottleneck. A 50-word sentence and a 5-word sentence had to fit into the same number of floating-point numbers. Their experiments showed it: the plain encoder-decoder got worse quickly as sentences got longer.
The key idea
Instead of one summary vector, keep one vector per source word and let the decoder choose which ones to read at each step. The paper calls these per-word vectors annotations. For each target word, the decoder scores every annotation, turns the scores into weights that sum to 1, and takes the weighted average. That average is the context vector for this step. The weights are learned along with everything else, so the model learns its own soft alignment between source and target words.
Figure 2. One decoder step of Bahdanau attention. The weights shown (0.80 on "agreement" while producing "accord") are illustrative; the scoring network and the weighted sum are from Section 3 of the paper.
Here is the maths, written plainly. Index source words with j and target words with i.
h_j = annotation for source word j (forward and backward RNN states, concatenated)
s_(i-1) = decoder hidden state before producing target word i
e_ij = v_a^T tanh( W_a s_(i-1) + U_a h_j ) # alignment score, a small neural net
alpha_ij = exp(e_ij) / sum_k exp(e_ik) # softmax over all source words
c_i = sum_j alpha_ij h_j # context vector for step i
W_a, U_a and v_a are learned weights. tanh squashes each number into the range minus 1 to 1. Because the score is computed by a small feed-forward network that adds two projected vectors, this is now called additive attention.
Architecture details as stated in the paper
- Task: English to French, WMT 2014 data, reduced to 348 million words with data selection. Each language used a shortlist of the 30,000 most frequent words; everything else became an unknown-word token.
- Encoder: a bidirectional RNN (one RNN reading left to right, one right to left), each direction with 1,000 hidden units, so each annotation joins two 1,000-wide states.
- Decoder: 1,000 hidden units. Word embeddings: 620 dimensions. The output layer used a single maxout hidden layer of 500 units. The alignment network's hidden layer had 1,000 units.
- Training: minibatch SGD with Adadelta, 80 sentences per minibatch, roughly 5 days per model. Translations were decoded with beam search.
- Two training lengths: sentences of up to 30 words (RNNsearch-30) and up to 50 words (RNNsearch-50), against plain encoder-decoders of the same sizes (RNNencdec-30 and RNNencdec-50).
Results
BLEU is the standard translation metric, an overlap score between machine and reference translations where higher is better. The paper's Table 1:
| Model | BLEU, all sentences | BLEU, no unknown words |
|---|---|---|
| RNNencdec-30 | 13.93 | 24.19 |
| RNNsearch-30 | 21.50 | 31.44 |
| RNNencdec-50 | 17.82 | 26.71 |
| RNNsearch-50 | 26.75 | 34.16 |
| RNNsearch-50, trained longer | 28.45 | 36.15 |
| Moses (phrase-based system) | 33.30 | 35.63 |
On sentences with no unknown words, the attention model matched and slightly beat Moses, the conventional phrase-based system, even though Moses used an extra 418 million words of monolingual text. On all sentences Moses still won, mostly because the neural model's 30,000-word vocabulary produced unknown tokens.
Why it mattered and what changed later
The jump from 17.82 to 26.75 BLEU at the same size, with no change except attention, made the case. The alignment plots in the paper also showed the model learning word order differences between English and French without being told. Within a year attention was standard in translation. Within three, the Transformer had dropped the RNN entirely and kept only attention. What later work changed: Luong simplified the score function, Vaswani et al. replaced the additive score with a scaled dot product that runs as one matrix multiply, and attention moved from connecting a decoder to an encoder (cross-attention) to connecting a sequence with itself (self-attention).
How did Luong attention differ from Bahdanau attention?
Luong, Pham and Manning (2015) simplified attention and tested variants side by side. They scored the current decoder state against source states with a plain dot product or a bilinear form, compared global attention over all words with local attention over a window, and set a new English to German record.
The paper
"Effective Approaches to Attention-based Neural Machine Translation" by Minh-Thang Luong, Hieu Pham and Christopher D. Manning, posted August 2015 (arXiv 1508.04025) and published at EMNLP 2015.
What they changed
Three things matter for the rest of this post.
First, the scoring function. Luong et al. compared several ways to score a target hidden state h_t against a source hidden state h_s:
dot: score(h_t, h_s) = h_t^T h_s
general: score(h_t, h_s) = h_t^T W_a h_s
concat: score(h_t, h_s) = v_a^T tanh( W_a [h_t ; h_s] ) # Bahdanau-style
location: a_t = softmax( W_a h_t ) # position only
The dot score is just a dot product: multiply matching entries and add them up. It has no parameters of its own. This is the direct ancestor of the Transformer's score. The authors reported that dot worked well for global attention and general was better for local attention, and that their concat implementation did not perform well.
Second, global versus local. Global attention looks at every source word, as Bahdanau did. Local attention first picks a position p_t in the source and only attends to a window of words around it, from p_t-D to p_t+D, with D = 10 chosen empirically. In the "local-m" variant p_t = t (assume the sentences line up); in "local-p" the model predicts p_t and favours nearby words with a Gaussian. This is an early version of the windowed attention we return to at the end of the post.
Third, the computation path. Bahdanau used the previous decoder state to compute attention; Luong used the current top-layer state, then combined the context with it through tanh(W_c [c_t ; h_t]) before predicting the word. Luong also fed each step's attentional vector into the next step's input ("input feeding") so the model knew what it had already attended to.
Architecture and results
The models were 4-layer stacked LSTMs (a type of RNN), each layer with 1,000 cells, 1,000-dimensional embeddings, and a 50,000-word vocabulary per language. Training used WMT 2014 English-German, 4.5 million sentence pairs. Local attention with predictive alignment beat non-attentional systems by 5.0 BLEU. An ensemble of 8 models with unknown-word replacement reached 23.0 BLEU on WMT 2014 English to German, and 25.9 BLEU on WMT 2015 English to German, which the authors reported as 1.0 BLEU above the previous best system.
Why it mattered
Luong's paper did two things the Transformer built on. It showed that a parameter-free dot product scores as well as a learned network, and dot products batch into one matrix multiply on a GPU. It also named the global versus local trade-off, which every long-context method since has revisited.
What is scaled dot-product attention?
Scaled dot-product attention computes, for each query, a dot product with every key, divides by the square root of the key dimension, applies a softmax to get weights that sum to 1, and returns the weighted sum of the values. In matrix form it is softmax(QK^T / sqrt(d_k)) V.
This is the form from "Attention Is All You Need" by Vaswani et al. (2017, arXiv 1706.03762), covered in depth in part 1. Here we take the formula apart.
The three roles: query, key, value
Every token's vector is projected three ways by learned weight matrices.
- Query (q): what this token is looking for.
- Key (k): what this token offers to be found by.
- Value (v): what this token hands over if it is chosen.
A useful analogy is a fuzzy dictionary lookup. A normal dictionary returns the value whose key exactly matches your query. Attention returns a blend of all values, weighted by how well each key matches the query.
Stacking all tokens' vectors gives matrices. With n tokens and model width d_model, the input X is n × d_model. Then:
Q = X W_Q shape n x d_k
K = X W_K shape n x d_k
V = X W_V shape n x d_v
Attention(Q, K, V) = softmax( Q K^T / sqrt(d_k) ) V shape n x d_v
K^T is K transposed (rows and columns swapped), so Q K^T is n × n: entry (i, j) is the dot product of query i with key j. The softmax runs along each row, turning that row of scores into weights. Multiplying by V gives each token a weighted mix of all value vectors. In the base Transformer, d_model = 512 and each head uses d_k = d_v = 64.
Figure 3. One attention head with tensor shapes. The n × n score and weight matrices are the part whose cost grows with the square of sequence length, and that fact drives everything in the second half of this post.
A 3-token example, by hand
Take three tokens, t1 to t3, with 4-dimensional queries and keys and 2-dimensional values. We skip the projections and write Q, K and V directly so the arithmetic stays small.
Q = [[1, 0, 1, 0], K = [[1, 0, 1, 0], V = [[1, 0],
[0, 2, 0, 2], [0, 1, 0, 1], [0, 1],
[1, 1, 1, 1]] [1, 1, 0, 0]] [1, 1]]
Step 1, scores. Entry (i, j) of Q K^T is q_i · k_j. For t1, q_1 = [1, 0, 1, 0]:
q_1 · k_1 = 1*1 + 0*0 + 1*1 + 0*0 = 2
q_1 · k_2 = 1*0 + 0*1 + 1*0 + 0*1 = 0
q_1 · k_3 = 1*1 + 0*1 + 1*0 + 0*0 = 1
Doing all nine gives:
S = Q K^T = [[2, 0, 1],
[0, 4, 2],
[2, 2, 2]]
Step 2, scale. d_k = 4, so divide by sqrt(4) = 2:
S / 2 = [[1.0, 0.0, 0.5],
[0.0, 2.0, 1.0],
[1.0, 1.0, 1.0]]
Step 3, softmax each row. The softmax of a list x is exp(x_j) / sum_k exp(x_k). For row t2, the exponentials are e^0 = 1, e^2 = 7.389 and e^1 = 2.718. They add up to 11.107, so the weights are 0.090, 0.665 and 0.245. Token t2 mostly attends to t2's own key. Doing every row:
P = [[0.507, 0.186, 0.307],
[0.090, 0.665, 0.245],
[0.333, 0.333, 0.333]]
Row t3 is uniform because its query [1, 1, 1, 1] has the same dot product (2) with all three keys.
Step 4, weighted sum of values.
out_t1 = 0.507*[1,0] + 0.186*[0,1] + 0.307*[1,1] = [0.814, 0.494]
out_t2 = 0.090*[1,0] + 0.665*[0,1] + 0.245*[1,1] = [0.335, 0.910]
out_t3 = 0.333*[1,0] + 0.333*[0,1] + 0.333*[1,1] = [0.667, 0.667]
Figure 4. The full computation for three tokens. Shaded weights are above 0.5. Every number can be checked with a calculator.
That is the whole operation. A real model does this with n in the thousands, d_k of 64 or 128, many heads in parallel and dozens of layers stacked, but nothing conceptually new is added.
Why dot products rather than Bahdanau's additive scores?
Vaswani et al. note that the two have similar theoretical complexity, but dot-product attention is "much faster and more space-efficient in practice" because it is a single matrix multiply, the operation GPUs are built for. Additive attention needs a hidden layer and a tanh for every query-key pair.
Why is attention scaled by the square root of d_k?
Without scaling, dot products grow with dimension: for random vectors with unit-variance entries, the dot product has variance d_k, so its typical size grows like sqrt(d_k). Large scores push the softmax into near one-hot outputs with vanishing gradients. Dividing by sqrt(d_k) brings the variance back to 1.
The variance argument
Footnote 4 of the Transformer paper gives the reasoning. Suppose each entry of q and k is an independent random number with mean 0 and variance 1. Then:
q · k = q_1 k_1 + q_2 k_2 + ... + q_dk k_dk
each term q_i k_i has mean 0 and variance 1 (product of independent unit-variance values)
sum of d_k independent terms: mean 0, variance d_k
standard deviation = sqrt(d_k)
(q · k) / sqrt(d_k) has variance d_k / d_k = 1
We checked this numerically for this post with 20,000 random vector pairs at each size. The measured standard deviation was 1.0 at d_k = 1, 8.0 at d_k = 64 and 22.4 at d_k = 512, against sqrt(512) = 22.6.
Why big scores hurt
Softmax is sensitive to the size of its inputs, not only their order. Take three scores [1, 0.5, 0] and the same scores multiplied by 8, which is sqrt(64):
softmax([1, 0.5, 0]) = [0.507, 0.307, 0.186]
softmax([8, 4, 0]) = [0.982, 0.018, 0.0003]
The second is almost one-hot: nearly all weight on one key. That is a problem for learning. The derivative of a softmax output with respect to its own input is p(1-p). For the top weight that is 0.25 in the first case and 0.018 in the second, about 14 times smaller, and the other entries are smaller still. The model gets almost no signal about how to change the scores. The paper's wording is that large dot products push the softmax "into regions where it has extremely small gradients".
Figure 5. Why scaling exists. Left: dot products of random vectors spread out as dimension grows. Right: the same ordering of scores, made 8 times larger, collapses the softmax onto one key.
Was the scaling tested?
The paper reports a related observation rather than an ablation: for small d_k, additive and dot-product attention performed similarly, but additive attention beat unscaled dot-product attention at larger d_k (citing earlier work). The scale factor is the authors' fix for that gap. In practice, almost every model since uses 1 / sqrt(d_k). Some later papers adjust or learn the factor, but the default has stuck.
How does causal masking work?
Causal masking stops each token from attending to tokens that come after it. Before the softmax, every score where the key position is later than the query position is set to minus infinity, so those weights become exactly zero. This lets a decoder train on whole sequences at once while still predicting left to right.
Why it is needed
A language model is trained to predict token t+1 from tokens 1 to t. During training, the whole sequence is processed in one pass for speed, so every token's query can see every key. Without a mask, token 3 could read token 4's value and copy the answer. The Transformer paper describes this as preventing "leftward information flow" in the decoder to keep it auto-regressive.
How it is implemented
Add a mask matrix M to the scaled scores before the softmax:
M[i][j] = 0 if j is at or before i (allowed)
M[i][j] = -inf if j is after i (blocked)
P = softmax( Q K^T / sqrt(d_k) + M )
Because exp(-inf) = 0, the blocked positions contribute nothing to the softmax's sum and get weight zero. The allowed weights in each row still add to 1.
Back to our 3-token example. With the mask, row t1 can only see k1, so its weight is [1, 0, 0] and its output is [1, 0]. Row t2 sees k1 and k2 with scaled scores 0 and 2; the softmax of [0, 2] is [0.119, 0.881], so its output becomes [0.119, 0.881] instead of [0.335, 0.910]. Row t3 can already see everything, so it is unchanged.
Figure 6. A causal mask over "Attention is all you need". The right panel shows the special case of equal allowed scores, where each row spreads its weight evenly over the tokens it may see.
Two practical notes
In code, implementations often use a large negative number rather than true infinity, which avoids NaN results when a whole row is masked. And there is a second kind of mask, the padding mask, which hides the filler tokens added so that sequences of different lengths fit in one batch. Encoder models such as BERT use only padding masks; decoder-only models such as GPT and Llama use both.
The mask also has a performance consequence. Roughly half the score matrix is masked, so a kernel that knows the mask can skip those blocks entirely. FlashAttention-2 does exactly that, which we come back to below.
What is multi-head attention and why use several heads?
Multi-head attention runs several smaller attention operations in parallel, each with its own learned projections, then concatenates their outputs and mixes them with one more matrix. The base Transformer splits a 512-wide model into 8 heads of 64, so each head can learn a different pattern at the same total cost.
The formula
head_i = Attention( X W_Q_i, X W_K_i, X W_V_i ) # each n x d_k
MultiHead(X) = Concat(head_1, ..., head_h) W_O # n x d_model
W_Q_i, W_K_i are d_model x d_k
W_V_i is d_model x d_v
W_O is (h * d_v) x d_model
In the base model, h = 8 and d_k = d_v = 512 / 8 = 64. Because each head works in 64 dimensions instead of 512, the total compute is similar to one full-width head. Counting the projection weights for one layer (ignoring biases): W_Q, W_K, W_V and W_O are each effectively 512 × 512, so 4 × 512 × 512 = 1,048,576 weights per attention layer.
Figure 7. Eight heads of 64 in a 512-wide model. Each head has its own projections and produces its own attention pattern; W_O mixes the eight outputs.
Why more than one head?
The paper's argument is short: one head averages, and averaging blurs. With a single softmax per position, a token that needs to look at both its subject and its object has to split one set of weights between them. Several heads let it attend to different positions for different reasons, in what the authors call different "representation subspaces".
They also tested it. Table 3, rows (A), varies the number of heads while keeping compute constant, on the English-German development set:
| Heads (h) | d_k = d_v | BLEU (dev) |
|---|---|---|
| 1 | 512 | 24.9 |
| 4 | 128 | 25.5 |
| 8 | 64 | 25.8 |
| 16 | 32 | 25.8 |
| 32 | 16 | 25.4 |
One head was 0.9 BLEU worse than the best setting, and quality dropped again with too many heads, because each head's dimension gets too small to carry much.
What do attention heads actually learn?
Studies of trained models find heads with recognisable jobs: attending to the previous token, to delimiters, to syntactic relations such as a verb's direct object, or to earlier copies of the current token. They also find that many heads can be removed with little loss, so a few specialised heads do most of the work.
This is an area where it is easy to overclaim, so here is only what the papers report.
BERT's heads have patterns (Clark et al., 2019). "What Does BERT Look At?" by Kevin Clark, Urvashi Khandelwal, Omer Levy and Christopher Manning (arXiv 1906.04341) studied the 144 attention heads of BERT-base. They found heads that attend to fixed positional offsets, heads that attend broadly, and a surprising amount of attention on the [SEP] delimiter token: over half of BERT's attention in layers 6 to 10 goes there, which they read as a kind of no-op, a place to put attention when a head has nothing useful to do. They also found individual heads that track direct objects of verbs, determiners of nouns, objects of prepositions and coreferent mentions with high accuracy.
Specialised heads do the heavy lifting (Voita et al., 2019). Elena Voita and colleagues (arXiv 1905.09418, ACL 2019) analysed translation Transformers and identified three recurring roles: positional heads that point to an adjacent token, syntactic heads that track specific dependency relations, and heads that point to rare words. Pruning showed how uneven the work is: on English-Russian WMT, removing 38 of the 48 encoder heads cost only 0.15 BLEU. The heads with clear positional and syntactic functions were the last to be pruned.
Many heads are redundant at test time (Michel et al., 2019). "Are Sixteen Heads Really Better than One?" by Paul Michel, Omer Levy and Graham Neubig (arXiv 1905.10650, NeurIPS 2019) found that a large share of heads can be removed after training without a significant drop, and that some layers can be cut to a single head.
Induction heads (Olsson et al., 2022). Researchers at Anthropic described "induction heads" (arXiv 2209.11895), heads that complete patterns of the form [A][B] ... [A] → [B]: if the sequence earlier contained token A followed by token B, and A appears again, the head attends to the B that followed the earlier A and boosts it as the next token. The paper presents what it calls preliminary and indirect evidence that these heads might be the mechanism behind most in-context learning in large Transformers.
A caution worth keeping. An attention weight tells you where a head read information from. It does not by itself tell you why the model produced its output, because later layers transform what was read. Attention maps are a good starting point for investigation, not an explanation.
The pruning results have a practical echo. If many heads carry redundant information, perhaps the keys and values do not need a separate copy per head. That is exactly the bet multi-query attention makes.
Why is generating text slow even on a fast GPU?
During generation the model produces one token at a time, and each new token's attention must read every earlier token's keys and values from GPU memory. That is little arithmetic per byte moved, so decoding is limited by memory bandwidth rather than compute. Every technique in the next sections attacks this bottleneck.
Training versus generation
During training, all n positions are processed in one pass. The big matrix multiplies have plenty of arithmetic for each number loaded from memory, which suits GPUs.
Generation is different. To produce token 1,001 the model runs one new query through every layer, and that query must attend to the keys and values of all 1,000 earlier tokens. The earlier keys and values do not change, so they are stored rather than recomputed. That store is the KV cache (next section). The arithmetic per step is small; the data that must be read is large.
Arithmetic intensity, worked through
Arithmetic intensity is the number of floating-point operations (FLOPs) done per byte read from memory. A chip is compute-bound when this number is high and memory-bound when it is low.
For one head, one new query, n cached tokens and head size d, in 16-bit precision (2 bytes per number):
FLOPs: q · k_j for all j = 2 * n * d (a multiply and an add per entry)
sum_j p_j v_j = 2 * n * d
total = 4 * n * d
Bytes: read K (n x d) and V (n x d) at 2 bytes each = 4 * n * d
Intensity = 4nd / 4nd = 1 FLOP per byte
Now compare with the hardware. The FlashAttention-2 paper gives an A100's peak as 312 TFLOPs/s for 16-bit matrix multiplies, and the FlashAttention paper gives its HBM bandwidth as 1.5 to 2.0 TB/s. Using the 1.5 TB/s figure, the A100 can do about 208 FLOPs in the time it reads one byte. Decoding attention offers it 1. The arithmetic units sit idle most of the time, waiting for keys and values to arrive.
Batching many requests together helps the weight matrices, because one read of a weight serves every sequence in the batch. It does not help attention over the cache, because each sequence has its own keys and values. Noam Shazeer's 2019 analysis puts this formally: for incremental multi-head decoding, the ratio of memory access to arithmetic is Θ(n/d + 1/b), where b is batch size. Batching shrinks the 1/b term. Only changing the attention design shrinks n/d.
What is the KV cache and how much memory does it use?
The KV cache stores every past token's key and value vectors, in every layer, so they are not recomputed at each step. Its size is 2 × layers × KV heads × head dimension × bytes per number × tokens. A LLaMA-7B-shaped model in 16-bit precision needs 512 KiB per token, or 2 GiB at 4,096 tokens.
The formula
For each token, each layer stores one key vector and one value vector per KV head.
bytes per token = 2 * n_layers * n_kv_heads * d_head * bytes_per_value
total bytes = bytes per token * tokens_in_context * batch_size
2 = one key and one value
bytes_per_value = 2 for fp16 or bf16, 1 for 8-bit
Worked examples
We use shapes published in the papers. The LLaMA paper (arXiv 2302.13971, Table 2) lists its 6.7B model as 32 layers, width 4,096, 32 heads (so a head dimension of 128), and its 65.2B model as 80 layers, width 8,192, 64 heads (head dimension 128). Mistral 7B (arXiv 2310.06825, Table 1) lists 32 layers, 32 query heads, 8 KV heads and a head dimension of 128.
7B shape, 32 KV heads:
2 * 32 * 32 * 128 * 2 bytes = 524,288 bytes = 512 KiB per token
4,096 tokens -> 2 GiB
32,768 tokens -> 16 GiB
65B shape, 64 KV heads:
2 * 80 * 64 * 128 * 2 = 2,621,440 bytes = 2,560 KiB per token
4,096 tokens -> 10 GiB
32,768 tokens -> 80 GiB
The vLLM paper (arXiv 2309.06180) gives its own example with the same arithmetic: OPT-13B needs 800 KB per token (2 × 5,120 hidden size × 40 layers × 2 bytes), and at OPT's maximum of 2,048 tokens a single request's cache "can be as much as 1.6 GB".
Now the uncomfortable comparison. The 6.7B model's weights in 16-bit precision take about 13.4 GB. A batch of 32 requests at 4,096 tokens each needs 32 × 2 GiB = 64 GiB of KV cache, almost five times the weights. On a serving box the cache, not the model, decides how many users you can handle at once.
Figure 8. KV cache for one long sequence. The GQA-8 and MQA rows apply those head counts to the 65B shape as a what-if, to isolate the effect of KV heads. The last row anticipates the sliding window section.
The formula shows the levers. You cannot cut layers or tokens without changing the model or the task. You can cut bytes per value (quantising the cache, covered in the inference post later in this series). And you can cut n_kv_heads. That last lever is what MQA and GQA pull.
What is multi-query attention?
Multi-query attention (MQA), proposed by Noam Shazeer in 2019, keeps many query heads but gives them a single shared key head and a single shared value head. The KV cache shrinks by the number of heads, decoding gets much faster, and quality drops only slightly.
The paper
"Fast Transformer Decoding: One Write-Head is All You Need" by Noam Shazeer, posted November 2019 (arXiv 1911.02150). It is a short single-author paper with an unusually direct analysis.
The problem and the idea
Shazeer's diagnosis is the one in the section above: incremental decoding is slow because of "the memory-bandwidth cost of repeatedly loading" the large key and value tensors. His fix is to keep h different query projections but share one key projection and one value projection across all heads. Each head still computes its own attention pattern, because its queries differ. They all read the same keys and values.
In the memory-to-compute ratio, the troublesome n/d term becomes n/(d·h), a factor of h smaller. In plain words, every byte of cached keys and values now serves h query heads instead of one, so arithmetic intensity rises by h.
Architecture as stated in the paper
The baseline is an encoder-decoder Transformer for WMT 2014 English-German: 6 layers, d_model = 1024, d_ff = 4096, h = 8, d_k = d_v = 128, learned positional embeddings, about 211 million parameters, trained 100,000 steps on a 32-core TPUv3 cluster. For the multi-query version the feed-forward width was raised from 4,096 to 5,440 so the parameter count matched. Local-attention variants restricted decoder self-attention to the current and previous 31 positions.
Results
Quality (Table 1, WMT 2014 English-German):
| Attention | ln(PPL) dev | BLEU dev | BLEU test, beam 1 / beam 4 |
|---|---|---|---|
| Multi-head | 1.424 | 26.7 | 27.7 / 28.4 |
| Multi-query | 1.439 | 26.5 | 27.5 / 28.5 |
| Multi-head, h = 1 | 1.518 | 25.8 | n/a |
| Multi-head, d_k = 16 | 1.513 | 25.8 | n/a |
Speed (Table 2, amortised TPUv2 microseconds per output token, sequence length 128):
| Attention | Training | Greedy decode, encoder + decoder | Beam-4, encoder + decoder |
|---|---|---|---|
| Multi-head | 13.2 | 1.7 + 46 | 2.0 + 203 |
| Multi-query | 13.0 | 1.5 + 3.8 | 1.6 + 32 |
Training cost was unchanged. The decoder's per-token cost fell from 46 to 3.8 microseconds with greedy decoding, roughly 12 times faster, and from 203 to 32 with beam search. Quality dipped slightly on the development set; on the test set with beam 4, multi-query actually had the highest score, 28.5. The cheaper alternatives of shrinking heads or head size lost far more quality. On the Billion-Word language modelling benchmark, perplexity went from 29.9 to 30.2.
Why it mattered and what later work changed
MQA showed that the cache, not the weights, was where decoding time went, and that most of it was unnecessary. Google's PaLM used it. But a single KV head is a big cut for a very large model, and training a model with MQA from scratch commits you to its quality ceiling. Grouped-query attention was the compromise.
What is grouped-query attention?
Grouped-query attention (GQA) splits the query heads into G groups and gives each group its own shared key and value head. With G equal to the number of heads it is ordinary multi-head attention; with G equal to 1 it is MQA. Intermediate values recover most of multi-head quality at close to MQA speed.
The paper
"GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints" by Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón and Sumit Sanghai, posted May 2023 (arXiv 2305.13245) and accepted at EMNLP 2023.
Two contributions
Uptraining. You do not need to train from scratch. Take an existing multi-head checkpoint, build each group's key head and value head by mean-pooling (averaging) the original heads in that group, then continue pre-training for a small fraction α of the original steps. The paper tested α = 0.05, which took about 600 TPUv3 chip-days for T5-XXL, and found mean pooling worked better than keeping one head or random initialisation.
Grouping. GQA-G uses G key-value heads. GQA-1 is MQA; GQA-H (one group per head) is multi-head. The authors argue the grouping matters more as models grow, because larger models have more heads, so MQA's cut from H heads to 1 is proportionally harsher. They applied GQA to decoder self-attention and cross-attention, but not to encoder self-attention, where all positions are computed in parallel and bandwidth is not the main bottleneck.
Figure 9. The only thing that changes across the three designs is how many key and value heads exist. The cache size, and the bytes read per decoding step, scale with that number.
Results
The experiments used T5.1.1 models. Table 1 of the paper, averaged across summarisation, translation and question answering datasets:
| Model | Time per sample (paper's units) | Average score |
|---|---|---|
| MHA, T5-Large | 0.37 | 46.0 |
| MHA, T5-XXL | 1.51 | 47.2 |
| MQA, T5-XXL (uptrained) | 0.24 | 46.6 |
| GQA-8, T5-XXL (uptrained) | 0.28 | 47.1 |
GQA-8 at XXL size came within 0.1 points of full multi-head XXL while running at about the speed of MQA and more than five times faster than multi-head XXL. Uptrained MQA XXL was both better and faster than multi-head Large. The paper selected 8 groups as a favourable middle ground, noting that going from MQA to a few groups adds only modest inference time.
Who uses it
Llama 2 (arXiv 2307.09288) adopted GQA for its 34B and 70B models "to improve inference scalability". Its appendix compares MQA and GQA with "8 KV projections" against a multi-head baseline at 30B parameters, and reports GQA performing comparably to multi-head on most tasks and better than MQA on average. Mistral 7B uses 32 query heads and 8 KV heads: four queries per key-value head. In Figure 8 that is why the Mistral 7B row costs a quarter of the 7B multi-head row at the same depth and head size.
What later work changed
GQA trades cache size for quality by sharing heads. DeepSeek-V2 (arXiv 2405.04434) went a different way with Multi-head Latent Attention (MLA), which compresses keys and values into a smaller latent vector that is cached instead. The paper reports reducing the KV cache by 93.3% compared with DeepSeek 67B. MLA deserves its own write-up; the point here is that the KV cache has become a design target in its own right.
How does FlashAttention speed up attention without approximating it?
FlashAttention computes exactly the same result as standard attention but never writes the n × n score matrix to the GPU's main memory. It splits Q, K and V into blocks that fit in fast on-chip memory, computes the softmax incrementally with running statistics, and fuses every step into one kernel. Fewer memory trips means faster and leaner attention.
The paper
"FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness" by Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra and Christopher Ré, posted May 2022 (arXiv 2205.14135) and published at NeurIPS 2022.
The problem: a GPU has fast small memory and slow big memory
The paper's key word is IO-aware: count the reads and writes between levels of memory, not just the FLOPs. On the A100 used in the paper:
- SRAM, on-chip memory: 192 KB per streaming multiprocessor, 108 of them (about 20 MB in total), bandwidth around 19 TB/s.
- HBM, the GPU's main memory: 40 to 80 GB, bandwidth 1.5 to 2.0 TB/s.
- CPU DRAM: over 1 TB, but only about 12.8 GB/s from the GPU's side.
Standard attention, as written in most frameworks, runs as separate steps. It reads Q and K from HBM, computes S = QK^T and writes S back to HBM. It reads S, computes the softmax P and writes P to HBM. It reads P and V and computes the output. The n × n matrices S and P make round trips through the slow memory. For n = 4,096, one head's S has 16.8 million entries, and a model has many heads and layers. The paper counts the HBM accesses of standard attention as Θ(Nd + N²), where N is sequence length and d head size. The N² term dominates at long lengths.
Many earlier efficient-attention papers reduced FLOPs by approximating attention. The FlashAttention authors point out that many of these did not deliver wall-clock speedups, because FLOPs were not the bottleneck. Memory traffic was.
Figure 10. Why attention is memory-bound and how tiling fixes it. Standard attention moves the n × n matrices through slow HBM; FlashAttention keeps each tile on chip and writes only the output.
The fix, explained simply
Think of a chef with a tiny countertop next to a large, slow walk-in fridge. The standard recipe chops all the vegetables, carries the full tray to the fridge, brings it back to season it, carries it back again, and so on. FlashAttention's recipe is to bring one small batch at a time to the countertop, do every step to it there, and only carry the finished dish back.
In attention terms, it uses two techniques.
Tiling. Split Q, K and V into blocks along the sequence. Load one block of K and V into SRAM, then loop over blocks of Q, computing the small tile S_ij = Q_i K_j^T on chip, using it, and discarding it. The full S never exists in HBM. The block sizes are set from the SRAM size M and head size d (in the paper, B_c = ceil(M / 4d) and B_r = min(ceil(M / 4d), d)).
The online softmax. Tiling has an obstacle. A softmax normally needs the whole row of scores, because each weight is divided by the sum over the row, and for numerical stability you also subtract the row's maximum. FlashAttention uses a known trick, the online softmax (Milakov and Gimelshein, 2018, arXiv 1805.02867): process the row in pieces, keep three running numbers per query row, and rescale when a larger maximum shows up.
m = running maximum of scores seen so far
l = running sum of exp(score - m)
o = running sum of exp(score - m) * value # unnormalised output
when a new block arrives with scores s and values v:
m_new = max(m, max(s))
c = exp(m - m_new) # rescale factor for old totals
l = l * c + sum( exp(s - m_new) )
o = o * c + sum( exp(s - m_new) * v )
m = m_new
final output = o / l
The online softmax with numbers
One query, four keys, processed as two blocks of two. Scores and scalar values:
block 1: scores [1, 3], values [10, 20]
block 2: scores [2, 4], values [30, 40]
After block 1:
m = 3
f = [e^(1-3), e^(3-3)] = [0.1353, 1]
l = 1.1353
o = 0.1353*10 + 1*20 = 21.353
Block 2 has a bigger maximum, 4, so the old totals are scaled by c = e^(3-4) = 0.3679:
l = 1.1353*0.3679 + (e^(2-4) + e^(4-4)) = 0.4177 + 1.1353 = 1.5530
o = 21.353*0.3679 + (0.1353*30 + 1*40) = 7.855 + 44.060 = 51.916
output = 51.916 / 1.5530 = 33.429
Doing the full softmax over [1, 3, 2, 4] at once gives weights 0.032, 0.237, 0.087 and 0.644, and 0.032*10 + 0.237*20 + 0.087*30 + 0.644*40 = 33.429. Same answer, without ever holding the whole row.
Figure 11. The trick that makes tiling possible. Each block updates three running numbers, and a rescale factor corrects earlier totals when a larger score arrives.
Recomputation in the backward pass. Training needs gradients, and the standard backward pass reads the stored S and P matrices. FlashAttention stores only the output and the per-row statistics m and l, then recomputes the needed tiles of S from Q and K during the backward pass. That is more FLOPs, but because it avoids reading N × N matrices from HBM, the backward pass is faster anyway.
Kernel fusion. All of these steps run in one GPU kernel, so intermediate results never leave the chip. The authors wrote it in CUDA to control memory access directly.
How much it saves, in the paper's own numbers
The paper proves FlashAttention needs Θ(N² d² / M) HBM accesses, against Θ(Nd + N²) for standard attention. For typical head sizes of 64 to 128 and SRAM around 100 KB, d² is many times smaller than M, so the saving is large. It also proves that no exact attention algorithm can asymptotically do better across all SRAM sizes. Memory use becomes linear in sequence length, because the N × N matrix is never stored.
Measured on GPT-2 medium attention (sequence 1,024, head dimension 64, 16 heads, batch 64, one A100, forward plus backward), from the paper's Figure 2:
| Metric | Standard attention | FlashAttention |
|---|---|---|
| Compute (GFLOPs) | 66.6 | 75.2 |
| HBM reads and writes (GB) | 40.3 | 4.4 |
| Runtime (ms) | 41.7 | 7.3 |
Figure 12. FlashAttention does about 13% more arithmetic, moves about 9 times less memory, and finishes about 5.7 times faster. Memory traffic, not arithmetic, set the runtime.
End to end, the paper reports training BERT-large (sequence 512) 15% faster than the MLPerf 1.1 training speed record, GPT-2 (sequence 1K) 3 times faster than the HuggingFace and Megatron-LM baselines, and the long-range arena benchmark (sequences 1K to 4K) 2.4 times faster. Longer context also became affordable: the paper reports the first Transformers to beat chance on the Path-X task (sequence 16K, 61.4% accuracy) and, with block-sparse FlashAttention, Path-256 (sequence 64K, 63.1%).
Why it mattered
FlashAttention changed what "efficient attention" means. Instead of changing the maths, which risks quality, it changed the schedule of memory reads and writes, which does not. It is a drop-in replacement for the standard kernel, and frameworks absorbed it quickly. A related line of work reached a similar memory result on a different path: Rabe and Staats (2021, arXiv 2112.05682) showed that self-attention can be computed in O(sqrt(n)) memory with a practical accelerator implementation, against the common belief that it needs O(n²).
What did FlashAttention-2 change?
FlashAttention-2 (Dao, 2023) keeps the same exact algorithm but organises the GPU work better: fewer non-matrix-multiply operations, parallelism across the sequence as well as across heads, and a different split of work between warps. It runs about twice as fast as FlashAttention and reaches 50 to 73% of an A100's theoretical peak in the forward pass.
The paper
"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning" by Tri Dao, posted July 2023 (arXiv 2307.08691), presented at ICLR 2024 per the Princeton research record.
The problem
FlashAttention fixed memory traffic, but its forward pass reached only 25 to 40% of the GPU's theoretical maximum FLOPs per second, and the backward pass only 25 to 35%. An optimised matrix multiply can reach up to 80 to 90%. The paper traced the gap to how work was divided between thread blocks and warps. (On an NVIDIA GPU, a warp is a group of 32 threads that execute together; a thread block is a group of warps scheduled onto one streaming multiprocessor.)
Three changes
1. Fewer non-matmul FLOPs. GPUs have special units (tensor cores) for matrix multiplies. On the A100, peak is 312 TFLOPs/s for 16-bit matmul but only 19.5 TFLOPs/s for non-matmul 32-bit operations, so each non-matmul FLOP costs about 16 times more. FlashAttention-2 rewrites the online softmax so the output is kept unscaled during the loop and divided by l once at the end, and stores only the log-sum-exp L = m + log(l) for the backward pass instead of both m and l.
2. Parallelism over sequence length. FlashAttention used one thread block per head per batch element. With long sequences and small batches that leaves many of the A100's 108 streaming multiprocessors idle. FlashAttention-2 also splits each head's work along the sequence, so more blocks run at once.
3. Better work split between warps. In FlashAttention's forward pass, K and V were split across 4 warps while every warp saw all of Q (the "split-K" scheme). Each warp then had to write partial results to shared memory and synchronise to add them up. FlashAttention-2 splits Q across warps instead and lets every warp see all of K and V. Each warp computes its own slice of the output with no communication.
It also exploits the causal mask: blocks that lie entirely above the diagonal are skipped, which the paper reports as a 1.7 to 1.8 times speedup over attention without the mask, and only one block per row needs the mask applied at all. It supports MQA and GQA directly by indexing shared heads rather than copying them.
Results
About 2 times faster than FlashAttention; up to 73% of theoretical maximum throughput in the forward pass and up to 63% in the backward pass on an A100; and end-to-end GPT-style training at up to 225 TFLOPs/s per A100, which the paper gives as 72% model FLOPs utilisation.
What came next
FlashAttention-3 (Shah, Bikshandi, Zhang, Thakkar, Ramani and Dao, 2024, arXiv 2407.08608) targets NVIDIA's Hopper GPUs, where FlashAttention-2 reached only 35% utilisation on the H100. Using asynchronous tensor cores and memory movement, interleaving the softmax with matrix multiplies, and FP8 low precision, it reports 1.5 to 2.0 times speedups, up to 740 TFLOPs/s in FP16 (75% utilisation) and close to 1.2 PFLOPs/s in FP8. The pattern is consistent: same maths, a schedule fitted more tightly to the hardware.
What is PagedAttention and why does vLLM use it?
PagedAttention stores each request's KV cache in small fixed-size blocks that can sit anywhere in GPU memory, tracked by a block table, the way an operating system pages virtual memory. It removes most wasted cache memory and lets requests share blocks, so a server fits more requests at once and serves 2 to 4 times more throughput.
The paper
"Efficient Memory Management for Large Language Model Serving with PagedAttention" by Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang and Ion Stoica, posted September 2023 (arXiv 2309.06180) and published at SOSP 2023. It introduced vLLM.
The problem: the cache was allocated like a fixed array
FlashAttention speeds up one attention call. PagedAttention addresses a different problem: how a server lays out the KV caches of many concurrent requests in memory.
Earlier serving systems stored each request's KV cache in one contiguous chunk, sized for the longest possible output because the real length is unknown in advance. That wastes memory three ways. Reserved slots hold space for tokens not yet generated. Internal fragmentation is space that will never be used because the request finishes early. External fragmentation is space between chunks that is too small for a new request. The authors measured this: in existing systems, only 20.4% to 38.2% of KV cache memory held actual token states.
The idea: virtual memory for the KV cache
Operating systems solved the same problem decades ago with paging: a program sees contiguous memory, but the physical pages can be anywhere, and a page table maps between them. PagedAttention applies that to the cache.
- The KV cache of each sequence is split into blocks of a fixed number of tokens. vLLM's default block size is 16 tokens.
- Each sequence has logical blocks, numbered in order, and a block table that maps each logical block to a physical block anywhere in GPU memory.
- A new physical block is allocated only when the previous one fills up. Waste per request is therefore limited to the unfilled part of its last block.
- The attention kernel reads keys and values block by block through the table, so the blocks do not need to be contiguous.
Sharing falls out of the design. If two sequences start with the same prompt (parallel samples from one request, beam search candidates, or a shared system prompt), their block tables can point to the same physical blocks. A reference count tracks how many sequences use each block. When a sequence needs to write into a shared block, the block is copied first, the copy-on-write technique that operating systems use for forked processes.
Figure 13. Top: logical blocks map to scattered physical blocks through a block table, and a shared prefix is stored once. Bottom: the share of KV cache memory holding real token states in the paper's experiment.
Results
In the paper's Figure 2, vLLM used 96.3% of its KV cache memory for actual token states, against 20.4%, 26.8% and 38.2% for three configurations of the Orca serving system. Sharing saved 6.1% to 9.8% of memory for parallel sampling and 37.6% to 55.2% for beam search on the Alpaca trace, and more on the ShareGPT trace. Overall, vLLM improved throughput of popular LLMs by 2 to 4 times over FasterTransformer and Orca at the same level of latency.
How it relates to FlashAttention
They solve different problems and are used together. FlashAttention decides how one attention computation moves data between HBM and SRAM. PagedAttention decides where in HBM the cached keys and values live, so that more requests fit. The inference post later in this series covers the serving side in more depth, including continuous batching.
What are sliding window and sparse attention?
Sliding window attention lets each token attend only to the previous W tokens, so cost grows linearly with length instead of quadratically. Sparse attention uses fixed patterns, such as a local window plus a few global or strided positions, to skip most query-key pairs. Stacking layers lets information still travel beyond one window.
This section is brief because part 9 of this series is about long context. The ideas matter here because they change which cells of the score matrix get computed at all.
Figure 14. Which query-key pairs each pattern computes, for 12 tokens. The patterns are schematic. At 12 tokens the savings look modest; at 32,768 tokens full causal attention computes about 537 million pairs per head while a 4,096-token window computes roughly a quarter of that.
Sparse Transformer (Child, Gray, Radford and Sutskever, 2019, arXiv 1904.10509). Factorised attention into patterns such as "strided" (attend to the local window and to every k-th earlier position) so that cost falls from O(n²) to O(n sqrt(n)). The authors trained networks with hundreds of layers on sequences tens of thousands of steps long, across images, audio and text.
Longformer (Beltagy, Peters and Cohan, 2020, arXiv 2004.05150). Combined a local sliding window with "global" attention on a few task-chosen tokens (for example the question tokens in question answering), which attend to everything and are attended to by everything. Attention cost scales linearly with sequence length. It was pretrained from RoBERTa with position embeddings extended to 4,096 tokens.
BigBird (Zaheer et al., 2020, arXiv 2007.14062, NeurIPS 2020). Three parts: a set of global tokens, a local window, and a few random connections per token. The authors proved the sparse pattern keeps the known theoretical properties of full attention and reported handling sequences up to 8 times longer than was possible on similar hardware.
Mistral 7B (Jiang et al., 2023, arXiv 2310.06825). Brought sliding window attention into a widely used decoder, combined with GQA. Each layer attends to the previous W = 4096 positions. Because layer k reads from layer k-1, which itself read W tokens back, information can travel W tokens per layer. With 32 layers the paper gives a theoretical attention span of about 131K tokens. The window also caps the cache: Mistral uses a rolling buffer cache of fixed size W, storing position i at slot i mod W and overwriting old entries. On a 32K-token sequence the paper reports this cuts cache memory by 8 times without hurting quality, which matches the arithmetic in Figure 8 (4 GiB down to 0.5 GiB). For a 16K sequence with W = 4096, changes to FlashAttention and xFormers gave a 2 times speedup over a vanilla attention baseline.
The trade-off to remember: a sliding window does not see distant tokens directly. Information from beyond the window must be carried forward through intermediate tokens' hidden states, layer by layer, and there is no guarantee every detail survives the trip. That is why the span is described as theoretical.
How do these techniques fit together in a modern LLM?
They stack, because each targets a different cost. GQA shrinks what is cached, FlashAttention shrinks how much memory traffic each attention call makes, PagedAttention packs the caches of many requests into memory without waste, and sliding windows cap how far back attention reaches.
| Technique | Year | Changes the maths? | What it saves | Where it acts |
|---|---|---|---|---|
| Scaled dot product | 2017 | It is the maths | Replaces additive scoring with one matmul | Training and inference |
| Causal mask | 2017 | Defines the task | Lets a decoder train on whole sequences | Training and prefill |
| Multi-head | 2017 | Yes | Lets heads learn separate patterns at equal cost | Everywhere |
| Multi-query (MQA) | 2019 | Yes, one KV head | KV cache and bandwidth, by a factor of h | Decoding |
| Grouped-query (GQA) | 2023 | Yes, G KV heads | KV cache by H/G with little quality loss | Decoding |
| FlashAttention 1, 2, 3 | 2022 to 2024 | No, exact | HBM traffic; memory linear in n | Training and prefill |
| PagedAttention | 2023 | No, exact | Wasted cache memory across requests | Serving |
| Sliding window | 2020, 2023 | Yes, limited reach | Compute and cache beyond W tokens | Long contexts |
A rough mental model of a single request on a modern open-weights model: during prefill, the prompt is processed in one pass; this step is compute-heavy and benefits most from FlashAttention. During decoding, one token at a time is produced; each step reads the whole KV cache, and this is where GQA, a smaller cache format and PagedAttention earn their keep. Across many users, the server's capacity is set mostly by how many KV caches fit in GPU memory at once.
The concession worth making: none of this is free. GQA and MQA cost a little quality, sliding windows trade direct long-range access for capacity, and kernels like FlashAttention are hardware-specific code that must be rewritten for each new GPU generation. Teams pick a point on each trade-off, and the papers above are where the numbers for those trade-offs come from.
What does this mean for securing what you build on these models?
The efficiency stack above is mostly native code, inference servers and model files pulled in as dependencies. Each layer is part of your software supply chain, and the risks sit in loading model weights, in the packages that serve them, and in the agents and tools that wrap them.
Everything in this post ships to you as software. FlashAttention is a compiled CUDA package. PagedAttention ships inside vLLM, an inference server with its own network listeners. Model weights arrive as files in formats such as pickle, safetensors, GGUF and ONNX. Each of those is something a team installs and runs, usually with broad permissions on a GPU box.
A few places where Safeguard helps, kept short because this post is about attention:
- Model file scanning. Safeguard checks model files in pickle, safetensors, GGUF and ONNX formats before they are loaded, flagging formats that can run code on load and unsafe deserialization patterns. Our posts on safetensors, pickle and GGUF and the model file extension risk table explain what each format can and cannot do.
- SCA with reachability across AI SDKs. Inference stacks pull in large dependency trees: serving frameworks, attention kernels, tokenizers and client SDKs. Safeguard's software composition analysis inventories them and uses reachability analysis to show whether your code actually calls the vulnerable function, so teams fix the vulnerabilities that matter first. Inference servers have had their share of deserialization advisories; see our round-up of pickle-over-network advisories in AI inference and how our reachability engine works.
- Guard runtime guardrails for AI agents. When a model drives tools, Guard applies runtime guardrails to what the agent does, including inspecting network egress for PII leaving the environment.
- MCP and connector security. Agents increasingly reach tools through MCP servers. Safeguard's MCP Guardrail proxies MCP traffic with policy checks; our enterprise MCP guide covers the wider set of controls, and our post on prompt injection defence architectures covers the model-facing side.
The attention maths is public and well understood. The code that runs it on your hardware is where the day-to-day risk lives.
Try it yourself
Click through these interactive Safeguard demos to see the steps described in this guide.
Test yourself
Why is generating text slow even on a fast GPU?
The model produces one token at a time, and each new token must read every earlier token's keys and values from GPU memory. That is little arithmetic per byte moved, so decoding is limited by memory bandwidth rather than compute.
What does the KV cache store, and how big can it get?
It stores every past token's key and value vectors in every layer, so they are not recomputed. A LLaMA-7B-shaped model in 16-bit precision needs 512 KiB per token, or 2 GiB at 4,096 tokens.
How does multi-query attention differ from multi-head attention?
MQA keeps many query heads but shares a single key head and a single value head. The KV cache shrinks by the number of heads and decoding gets much faster, with only a slight drop in quality.
Where does grouped-query attention sit between those two?
GQA splits the query heads into G groups and gives each group its own shared key and value head. With G equal to the number of heads it is multi-head attention, and with G equal to 1 it is MQA.
How does FlashAttention speed things up without approximating?
It computes exactly the same result but never writes the n by n score matrix to main GPU memory. It works in blocks that fit in fast on-chip memory, computes softmax incrementally, and fuses the steps into one kernel.
What problem does PagedAttention solve?
Older systems stored each request's KV cache in one contiguous chunk sized for the longest output, and only 20.4 to 38.2 percent of that memory held real token states. PagedAttention uses small blocks tracked by a block table, so a server fits more requests at once.
FAQ
What is attention in a large language model?
Attention is the step where each token gathers information from other tokens. Each token's vector is projected into a query, a key and a value. The query is compared with every key by dot product, the scores are scaled and turned into weights by a softmax, and the token's output is the weighted average of the values. In a decoder, a causal mask limits this to earlier tokens.
Why is attention divided by the square root of d_k?
If query and key entries have variance 1, their dot product has variance d_k, so scores grow with head size. Large scores make the softmax nearly one-hot, and its gradients become very small, which slows learning. Dividing by sqrt(d_k) returns the scores to variance 1. In this post's example, scaling the scores 1, 0.5 and 0 by 8 moved the top weight from 0.507 to 0.982.
What is the difference between multi-head, multi-query and grouped-query attention?
All three use many query heads. Multi-head attention gives every query head its own key and value head. Multi-query attention (Shazeer, 2019) shares one key and value head across all query heads. Grouped-query attention (Ainslie et al., 2023) shares one key and value head per group of query heads. Fewer key and value heads means a smaller KV cache and faster decoding, at a small cost in quality.
How much memory does the KV cache use?
Multiply 2 (key and value) by the number of layers, the number of KV heads, the head dimension, the bytes per number and the number of tokens. A model with 32 layers, 32 KV heads of size 128 in 16-bit precision needs 512 KiB per token, so 2 GiB for a 4,096-token context and 16 GiB for 32,768 tokens, per sequence. Multiply by batch size for concurrent requests.
Is FlashAttention an approximation of attention?
No. FlashAttention computes exact attention, the same result as the standard formula up to floating-point rounding. It changes how the computation is scheduled on the GPU: it processes blocks in fast on-chip memory, uses an online softmax to combine blocks correctly, and avoids writing the n × n score matrix to slow memory. Block-sparse FlashAttention, an extension in the same paper, is the approximate variant.
What is the difference between FlashAttention and PagedAttention?
FlashAttention is a GPU kernel that makes one attention computation faster by cutting memory traffic. PagedAttention is a memory management scheme for serving: it stores each request's KV cache in fixed-size blocks mapped through a block table, which removes fragmentation and lets requests share cached prefixes. Serving systems such as vLLM use both together.
What does causal masking do?
It stops a token from attending to later tokens. Scores for future positions are set to minus infinity before the softmax, which makes their weights exactly zero. This lets a decoder train on a whole sequence in one pass without seeing the answers, and it lets optimised kernels skip the masked half of the score matrix.
Does sliding window attention lose information outside the window?
Not entirely, but it can. Each layer only attends directly to the previous W tokens, yet information moves up to W tokens further back with every layer, because each token's hidden state already summarises its own window. Mistral 7B's paper gives a theoretical span of about 131K tokens from a 4,096 window and 32 layers. Whether a specific distant detail survives that path is not guaranteed.
Sources
- Bahdanau, Cho, Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015. arXiv 1409.0473
- Luong, Pham, Manning. Effective Approaches to Attention-based Neural Machine Translation. EMNLP 2015. arXiv 1508.04025
- Vaswani et al. Attention Is All You Need. NeurIPS 2017. arXiv 1706.03762
- Shazeer. Fast Transformer Decoding: One Write-Head is All You Need. 2019. arXiv 1911.02150
- Ainslie et al. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. EMNLP 2023. arXiv 2305.13245
- Dao, Fu, Ermon, Rudra, Ré. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS 2022. arXiv 2205.14135
- Dao. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. ICLR 2024. arXiv 2307.08691
- Shah et al. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision. 2024. arXiv 2407.08608
- Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. arXiv 2309.06180
- Milakov, Gimelshein. Online normalizer calculation for softmax. 2018. arXiv 1805.02867
- Rabe, Staats. Self-attention Does Not Need O(n^2) Memory. 2021. arXiv 2112.05682
- Touvron et al. LLaMA: Open and Efficient Foundation Language Models. 2023. arXiv 2302.13971
- Touvron et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. 2023. arXiv 2307.09288
- Jiang et al. Mistral 7B. 2023. arXiv 2310.06825
- DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. 2024. arXiv 2405.04434
- Child, Gray, Radford, Sutskever. Generating Long Sequences with Sparse Transformers. 2019. arXiv 1904.10509
- Beltagy, Peters, Cohan. Longformer: The Long-Document Transformer. 2020. arXiv 2004.05150
- Zaheer et al. Big Bird: Transformers for Longer Sequences. NeurIPS 2020. arXiv 2007.14062
- Clark, Khandelwal, Levy, Manning. What Does BERT Look At? An Analysis of BERT's Attention. 2019. arXiv 1906.04341
- Voita, Talbot, Moiseev, Sennrich, Titov. Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. ACL 2019. arXiv 1905.09418
- Michel, Levy, Neubig. Are Sixteen Heads Really Better than One? NeurIPS 2019. arXiv 1905.10650
- Olsson et al. In-context Learning and Induction Heads. 2022. arXiv 2209.11895