Transformers, Explained Without the Hype

A mental model that survives contact with code

A transformer is a stack of identical blocks operating on a running buffer of vectors — one vector per token. Each block reads the buffer, computes something, and adds its result back. Nothing is ever overwritten. Hold onto that picture and the architecture diagrams stop being intimidating.

1. The residual stream

Every block's output is added to its input: x ← x + f(x). That buffer is the residual stream. Attention layers move information between token positions; feed-forward layers transform information within a position. Both write into the same shared stream, which is why gradients flow so cleanly through very deep stacks.

2. Self-attention, in one formula

Each token projects itself into three roles: a query (what am I looking for?), a key (what do I offer?), and a value (what do I pass on?). Dot every query against every key, scale by √d_k so the softmax does not saturate, normalise into weights, and take a weighted average of the values. That is the whole mechanism.

# Single-head self-attention on four tokens, in pure Python.
from math import exp, sqrt

def softmax(xs):
    m = max(xs)
    exps = [exp(x - m) for x in xs]
    total = sum(exps)
    return [e / total for e in exps]

def matmul_vec(matrix, vec):
    return [sum(a * b for a, b in zip(row, vec)) for row in matrix]

tokens = ["the", "cat", "sat", "down"]
# Toy 3-dim embeddings, already projected into Q, K, V for brevity.
Q = [[1, 0, 0], [0, 1, 0], [0, 1, 1], [1, 0, 1]]
K = [[1, 0, 0], [0, 1, 0], [0, 1, 1], [1, 0, 1]]
V = [[1, 0, 0], [0, 2, 0], [0, 0, 3], [4, 0, 0]]
d_k = 3

for i, q in enumerate(Q):
    scores = [sum(a * b for a, b in zip(q, k)) / sqrt(d_k) for k in K]
    weights = softmax(scores)
    out = [sum(w * v[j] for w, v in zip(weights, V)) for j in range(3)]
    print("%-5s attends to %s" % (tokens[i], " ".join("%s=%.2f" % (t, w) for t, w in zip(tokens, weights))))
    print("      output: %s" % [round(o, 3) for o in out])

3. Why multiple heads

One attention pattern per layer would be a severe bottleneck: a token often needs to track its syntactic parent, the topic of the paragraph and the previous mention of an entity simultaneously. Multi-head attention splits the embedding into h slices and runs an independent attention pattern in each, then concatenates. Same cost, many more relationships.

4. Position, because attention has none

Self-attention is permutation-equivariant — shuffle the tokens and you shuffle the outputs identically. Order has to be injected. The original paper added fixed sinusoids to the embeddings; modern models overwhelmingly use rotary embeddings (RoPE), which rotate query and key vectors by an angle proportional to position so that the dot product depends only on relative distance.

from math import sin, cos, pow

def sinusoidal(pos, d_model=8):
    vec = []
    for i in range(0, d_model, 2):
        freq = 1.0 / pow(10000, i / d_model)
        vec.append(sin(pos * freq))
        vec.append(cos(pos * freq))
    return vec

for pos in range(4):
    print("pos=%d -> %s" % (pos, [round(v, 3) for v in sinusoidal(pos)]))
print()
print("Nearby positions get similar vectors; distant ones diverge — that is the whole trick.")

5. The feed-forward layer is where the facts live

Two thirds of a transformer's parameters sit in the position-wise MLP, not in attention. It expands each vector to roughly four times the model width, applies a non-linearity, and projects back. Interpretability work increasingly treats this layer as a key-value memory: attention decides what to look at, the MLP decides what that means.

6. Normalisation, masking and the rest

• Pre-norm (LayerNorm or RMSNorm before each sub-layer) is now standard — it makes deep stacks trainable without careful warm-up.
• Causal masking sets attention to future positions at −∞ before the softmax, so a decoder cannot read ahead.
• Encoder-only models (BERT-style) see the whole sequence and suit classification and embeddings.
• Decoder-only models (GPT-style) are causal and suit generation — the dominant design today.
• Attention cost grows quadratically with sequence length, which is why long-context work focuses on approximations, caching and sparsity.

What to remember

A transformer block is: attend (mix across positions), then think (transform within a position), both written into a shared residual stream, repeated N times. Everything else — head counts, normalisation placement, positional scheme — is engineering around that single idea.