ML reading group

Transformer &
self-attention

An attention-only sequence model replaces recurrence, adds positional encoding, and trains in parallel.

Attention weights relevance

No recurrence

Encoder + decoder

1 / 9

attention-is-all-you-need

The Transformer’s claim: self-attention is enough for sequence modeling.

The paper replaces recurrent and convolutional layers with stacked attention plus position-wise feed-forward networks, aiming for better parallelism and shorter paths between distant tokens.

Why it mattered

Higher training parallelism

Constant-depth paths within a layer

Strong translation quality with less compute

2 / 9

summary

Before

Recurrent / convolutional sequence models

Process tokens with more sequential steps
Long-range dependencies take longer paths
Parallelization is limited by the architecture

After

Transformer stacks of attention

All positions can interact in parallel
Self-attention shortens dependency paths
Encoder and decoder both reuse attention blocks

3 / 9

transformer-models

Attention in one line

query × keys → weighted values

A query is scored against keys, the scores become weights, and the values are combined into a context vector. Self-attention is the special case where query, key, and value all come from the same sequence.

4 / 9

attention-mechanisms

Scaled dot-product attention keeps scores from getting too sharp.

The paper scales dot products by the square root of key dimension before softmax. That prevents large raw scores from collapsing the distribution and makes training more stable.

Mechanism

Q

Query

→

K,V

Keys + values

Score, scale, softmax, then sum the values.

5 / 9

summary

Without recurrence or convolution, the model needs explicit position information; fixed sinusoidal encodings give each token a location while preserving parallel computation.

— positional encoding in the Transformer

6 / 9

positional-encoding

Encoder

6 identical layers
Multi-head self-attention
Position-wise feed-forward network

Decoder

6 identical layers
Masked self-attention + encoder attention
Residual connections and layer norm

7 / 9

architecture

Headline result

28.4 BLEU → 41.8 BLEU

The big model reached state-of-the-art translation results on WMT 2014 English-German and English-French, while also training faster thanks to more parallel computation.

8 / 9

results

Read the paper
with the attention map in mind.

When you see a Transformer block, ask three questions: what is being attended to, where does order come from, and how does the stack keep information flowing across the sequence?

9 / 9

thanks