Why We Need Attention

Before Transformer, the dominant approach to sequence processing was RNN and LSTM. They processed tokens one at a time — the third token must wait for the second token to complete.

This “must wait” created two fatal problems: no parallel training, and early information gets forgotten in long sequences.

In 2017, Vaswani et al.’s paper proposed a radical idea: abandon RNN entirely and use only attention. This allows the entire sequence to be processed simultaneously, making parallel training possible. For a deeper reading of the paper, see Reading Notes: Attention Is All You Need.

Self-Attention in Four Steps

Take the sentence “The cat sat on the mat”. When the model processes the word “sat”:

Step 1: Create Q, K, V

Each input token embedding is projected through three different weight matrices:

Q = input @ W_Q  # Query: What am I looking for?
K = input @ W_K  # Key: What information do I have?
V = input @ W_V  # Value: What is my actual content?

Step 2: Compute Attention Scores

scores = Q @ K.T  # shape: (seq_len, seq_len)
# scores[i][j] = how much token i attends to token j

Step 3: Scale + Softmax

d_k = Q.shape[-1]
attention_weights = softmax(scores / sqrt(d_k), dim=-1)

Step 4: Weighted Sum

output = attention_weights @ V

Complete Implementation

import torch
import torch.nn.functional as F

def self_attention(X, W_Q, W_K, W_V):
    Q, K, V = X @ W_Q, X @ W_K, X @ W_V
    d_k = Q.shape[-1]
    scores = Q @ K.T / torch.sqrt(torch.tensor(d_k))
    weights = F.softmax(scores, dim=-1)
    return weights @ V, weights

The attention map — a (6, 6) matrix where row i, column j shows how much token i attends to token j — is what people call the “attention heatmap”.

Multi-Head Attention

Multiple attention heads run in parallel, each attending to different feature subspaces. GPT’s every generation is built on the Transformer architecture, detailed in GPT Series Evolution. For inference optimization, see LLM Inference Optimization.