Reading Notes: Attention Is All You Need

Paper Background

In 2017, Vaswani et al. published a paper of just over 6,000 words that completely transformed NLP and the broader AI field.

Core Innovation

The paper’s key innovation was replacing RNNs entirely with Self-Attention. RNNs can’t parallelize; Self-Attention processes the entire sequence at once.

The Key Formula

$Attention(Q, K, V) = softmax(\frac{QK^T}{\sqrt{d_k}})V$

The scaling factor $\sqrt{d_k}$ prevents dot products from growing too large, which would cause vanishing gradients — a masterful engineering detail.

Why It Was So Impactful

Transformer proved that attention alone can work — no RNN or CNN needed. This paved the way for BERT, the GPT series, and everything that followed. More technical details in Transformer Attention Mechanism Explained.