50 · Full Transformer from Scratch#
Dependencies: torch (GPU rec.)
Runtime: ~10 min T4
Chapter stub — implementation coming in a future commit.
Time-machine sentences#
Every chapter runs the same 20 held-out test sentences so the quality arc across chapters is directly comparable.
# Colab setup: install this part's pinned dependencies (skipped outside Colab).
import sys
if "google.colab" in sys.modules:
get_ipython().system("wget -q -O requirements_part5.txt https://raw.githubusercontent.com/eduardosanchezg/mthistory/main/requirements_part5.txt")
get_ipython().system("pip install -q -r requirements_part5.txt")
The Transformer from Scratch (Vaswani et al. 2017)#
“Attention Is All You Need.” – Vaswani et al., NeurIPS 2017.
The 2017 Transformer replaced recurrence entirely with self-attention. No more sequential bottleneck: every position attends to every other position in a single parallel matrix multiply. This chapter builds every piece from scratch – scaled dot-product attention, multi-head attention, sinusoidal positional encodings, and layer normalisation – in pure numpy so the math actually runs. The full trainable model is then assembled in PyTorch (guarded, since this environment has no GPU).
Tiny config used throughout: d_model=256, heads=4, layers=3, ffn=512.
1. Setup: TRAIN_FROM_SCRATCH flag + torch guard#
As in every neural chapter we default to loading a checkpoint. Set
TRAIN_FROM_SCRATCH = True only if you have a GPU and ~10 minutes.
TRAIN_FROM_SCRATCH = False # set True if you have GPU / time
import numpy as np
np.random.seed(0)
try:
import torch
import torch.nn as nn
TORCH = True
except ImportError:
TORCH = False
print("PyTorch not installed -- showing architecture + pre-computed outputs.")
CKPT = "../../checkpoints/transformer_tiny_multi30k_en_de.pt"
# Tiny config (fits a 30-min free Colab T4 session on Multi30k)
D_MODEL = 256
N_HEADS = 4
N_LAYERS = 3
D_FF = 512
print("config: d_model=%d heads=%d layers=%d ffn=%d" % (D_MODEL, N_HEADS, N_LAYERS, D_FF))
2. Scaled dot-product attention (numpy)#
The core operation. For queries Q, keys K, values V:
\[\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V\]
The sqrt(d_k) scaling keeps the dot products from growing large (which would
push softmax into regions with vanishing gradients).
def softmax(x, axis=-1):
x = x - np.max(x, axis=axis, keepdims=True)
e = np.exp(x)
return e / np.sum(e, axis=axis, keepdims=True)
def scaled_dot_product_attention(Q, K, V, mask=None):
# Q: (..., seq_q, d_k) K: (..., seq_k, d_k) V: (..., seq_k, d_v)
d_k = Q.shape[-1]
scores = np.matmul(Q, np.swapaxes(K, -1, -2)) / np.sqrt(d_k)
if mask is not None:
scores = np.where(mask, scores, -1e9)
weights = softmax(scores, axis=-1)
output = np.matmul(weights, V)
return output, weights
# small demo
seq_q, seq_k, d_k, d_v = 3, 4, 8, 8
Q = np.random.randn(seq_q, d_k)
K = np.random.randn(seq_k, d_k)
V = np.random.randn(seq_k, d_v)
out, w = scaled_dot_product_attention(Q, K, V)
print("output shape:", out.shape, " (seq_q, d_v)")
print("weights shape:", w.shape, " (seq_q, seq_k)")
print("attention weights (each row sums to 1):")
print(np.round(w, 3))
print("row sums:", np.round(w.sum(axis=1), 3))
3. Multi-head attention (numpy)#
Instead of one attention with d_model-dim Q/K/V, we split into h heads of
size d_model/h, run attention in parallel per head, then concatenate and
project. Each head can specialise on a different relation.
def multi_head_attention(X_q, X_kv, Wq, Wk, Wv, Wo, n_heads):
# X_q: (seq_q, d_model) X_kv: (seq_k, d_model)
seq_q, d_model = X_q.shape
seq_k = X_kv.shape[0]
d_head = d_model // n_heads
Q = X_q @ Wq # (seq_q, d_model)
K = X_kv @ Wk
V = X_kv @ Wv
# split into heads: (n_heads, seq, d_head)
def split(t, seq):
return t.reshape(seq, n_heads, d_head).transpose(1, 0, 2)
Qh, Kh, Vh = split(Q, seq_q), split(K, seq_k), split(V, seq_k)
out_h, w_h = scaled_dot_product_attention(Qh, Kh, Vh) # (n_heads, seq_q, d_head)
# concat heads back to (seq_q, d_model)
concat = out_h.transpose(1, 0, 2).reshape(seq_q, d_model)
return concat @ Wo, w_h
d_model, n_heads, seq = 16, 4, 5
X = np.random.randn(seq, d_model)
Wq = np.random.randn(d_model, d_model) * 0.1
Wk = np.random.randn(d_model, d_model) * 0.1
Wv = np.random.randn(d_model, d_model) * 0.1
Wo = np.random.randn(d_model, d_model) * 0.1
mha_out, head_w = multi_head_attention(X, X, Wq, Wk, Wv, Wo, n_heads)
print("d_model=%d split into %d heads of size %d" % (d_model, n_heads, d_model // n_heads))
print("multi-head output shape:", mha_out.shape, " (seq, d_model)")
print("per-head weight tensor:", head_w.shape, " (n_heads, seq, seq)")
print("head 0 attention weights:")
print(np.round(head_w[0], 2))
4. Sinusoidal positional encoding (numpy)#
Self-attention is permutation-invariant – it has no notion of order. We inject position with fixed sinusoids:
\[PE(pos, 2i) = \sin\!\left(\frac{pos}{10000^{2i/d}}\right), \qquad
PE(pos, 2i+1) = \cos\!\left(\frac{pos}{10000^{2i/d}}\right)\]
Different frequencies per dimension let the model attend by relative offsets.
def positional_encoding(seq_len, d_model):
pe = np.zeros((seq_len, d_model))
pos = np.arange(seq_len)[:, None]
i = np.arange(0, d_model, 2)[None, :]
div = np.power(10000.0, i / d_model)
pe[:, 0::2] = np.sin(pos / div)
pe[:, 1::2] = np.cos(pos / div)
return pe
pe = positional_encoding(seq_len=10, d_model=16)
print("PE shape:", pe.shape)
print("first 4 dims of positions 0..4:")
print(np.round(pe[:5, :4], 3))
# nearby positions have more similar encodings (higher dot-product similarity)
def cos_sim(a, b):
return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
print()
print("similarity of position 0 to positions 0..9 (ASCII bar):")
for p in range(10):
s = cos_sim(pe[0], pe[p])
bar = "#" * int(round((s + 1) / 2 * 40))
print("pos %2d %+.3f %s" % (p, s, bar))
5. Feed-forward, layer norm, residual (numpy)#
Each sub-layer is wrapped as LayerNorm(x + Sublayer(x)) – a residual
connection followed by layer normalisation. LayerNorm standardises each
feature vector independently:
\[\mathrm{LN}(x) = \gamma \cdot \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta\]
The position-wise feed-forward network is two linear layers with a ReLU:
FFN(x) = ReLU(x W1 + b1) W2 + b2, expanding to d_ff then back to d_model.
def layer_norm(x, gamma=None, beta=None, eps=1e-5):
mu = x.mean(axis=-1, keepdims=True)
var = x.var(axis=-1, keepdims=True)
norm = (x - mu) / np.sqrt(var + eps)
if gamma is None:
gamma = np.ones(x.shape[-1])
if beta is None:
beta = np.zeros(x.shape[-1])
return gamma * norm + beta
def feed_forward(x, W1, b1, W2, b2):
h = np.maximum(0.0, x @ W1 + b1) # ReLU
return h @ W2 + b2
def sublayer_residual_norm(x, sublayer_out):
return layer_norm(x + sublayer_out)
seq, d_model, d_ff = 5, 16, 32
x = np.random.randn(seq, d_model) * 3 + 1
W1 = np.random.randn(d_model, d_ff) * 0.1; b1 = np.zeros(d_ff)
W2 = np.random.randn(d_ff, d_model) * 0.1; b2 = np.zeros(d_model)
ffn_out = feed_forward(x, W1, b1, W2, b2)
y = sublayer_residual_norm(x, ffn_out)
print("input per-row mean/std:", np.round(x.mean(1), 2)[:3], np.round(x.std(1), 2)[:3])
print("after residual+LN mean ~0:", np.round(y.mean(1), 4))
print("after residual+LN std ~1:", np.round(y.std(1), 4))
6. Full Transformer in PyTorch (guarded)#
Now we assemble the trainable model: an encoder stack of EncoderLayers and a
decoder stack of DecoderLayers (with masked self-attention + cross-attention).
If torch is unavailable we print the architecture as a labeled text tree.
if TORCH:
class MHA(nn.Module):
def __init__(self, d_model, n_heads):
super().__init__()
self.attn = nn.MultiheadAttention(d_model, n_heads, batch_first=True)
def forward(self, q, k, v, attn_mask=None, key_padding_mask=None):
o, _ = self.attn(q, k, v, attn_mask=attn_mask,
key_padding_mask=key_padding_mask)
return o
class EncoderLayer(nn.Module):
def __init__(self, d_model, n_heads, d_ff):
super().__init__()
self.self_attn = MHA(d_model, n_heads)
self.ff = nn.Sequential(nn.Linear(d_model, d_ff), nn.ReLU(),
nn.Linear(d_ff, d_model))
self.n1 = nn.LayerNorm(d_model)
self.n2 = nn.LayerNorm(d_model)
def forward(self, x, src_key_padding_mask=None):
x = self.n1(x + self.self_attn(x, x, x, key_padding_mask=src_key_padding_mask))
x = self.n2(x + self.ff(x))
return x
class DecoderLayer(nn.Module):
def __init__(self, d_model, n_heads, d_ff):
super().__init__()
self.self_attn = MHA(d_model, n_heads)
self.cross_attn = MHA(d_model, n_heads)
self.ff = nn.Sequential(nn.Linear(d_model, d_ff), nn.ReLU(),
nn.Linear(d_ff, d_model))
self.n1 = nn.LayerNorm(d_model)
self.n2 = nn.LayerNorm(d_model)
self.n3 = nn.LayerNorm(d_model)
def forward(self, x, memory, tgt_mask=None, mem_key_padding_mask=None):
x = self.n1(x + self.self_attn(x, x, x, attn_mask=tgt_mask))
x = self.n2(x + self.cross_attn(x, memory, memory,
key_padding_mask=mem_key_padding_mask))
x = self.n3(x + self.ff(x))
return x
class TinyTransformer(nn.Module):
def __init__(self, src_vocab, tgt_vocab, d_model=D_MODEL, n_heads=N_HEADS,
n_layers=N_LAYERS, d_ff=D_FF, max_len=64):
super().__init__()
self.src_emb = nn.Embedding(src_vocab, d_model)
self.tgt_emb = nn.Embedding(tgt_vocab, d_model)
pe = torch.tensor(positional_encoding(max_len, d_model), dtype=torch.float32)
self.register_buffer("pe", pe)
self.enc = nn.ModuleList([EncoderLayer(d_model, n_heads, d_ff)
for _ in range(n_layers)])
self.dec = nn.ModuleList([DecoderLayer(d_model, n_heads, d_ff)
for _ in range(n_layers)])
self.out = nn.Linear(d_model, tgt_vocab)
def encode(self, src, src_pad=None):
x = self.src_emb(src) + self.pe[:src.size(1)]
for l in self.enc:
x = l(x, src_key_padding_mask=src_pad)
return x
def decode(self, tgt, memory, tgt_mask=None, mem_pad=None):
x = self.tgt_emb(tgt) + self.pe[:tgt.size(1)]
for l in self.dec:
x = l(x, memory, tgt_mask=tgt_mask, mem_key_padding_mask=mem_pad)
return self.out(x)
def forward(self, src, tgt, tgt_mask=None, src_pad=None):
mem = self.encode(src, src_pad)
return self.decode(tgt, mem, tgt_mask, src_pad)
model = TinyTransformer(src_vocab=5000, tgt_vocab=5000)
n_params = sum(p.numel() for p in model.parameters())
print("TinyTransformer built. parameters: %.2fM" % (n_params / 1e6))
else:
tree = [
"TinyTransformer (Multi30k EN->DE)",
" src_embedding: Embedding(5000 -> 256)",
" tgt_embedding: Embedding(5000 -> 256)",
" positional_encoding: sinusoidal, max_len=64, d=256",
" Encoder x3:",
" EncoderLayer:",
" self_attn: MultiHead(d_model=256, heads=4, d_head=64)",
" LayerNorm + residual",
" feed_forward: Linear(256->512) ReLU Linear(512->256)",
" LayerNorm + residual",
" Decoder x3:",
" DecoderLayer:",
" masked_self_attn: MultiHead(256, 4) [causal mask]",
" LayerNorm + residual",
" cross_attn: MultiHead(256, 4) [attends to encoder memory]",
" LayerNorm + residual",
" feed_forward: Linear(256->512) ReLU Linear(512->256)",
" LayerNorm + residual",
" output_proj: Linear(256 -> 5000)",
"",
"approx parameters: ~7.0M",
]
print("\n".join(tree))
7. Synthetic warmup task: number-to-words#
Before Multi30k we warm up on a trivial deterministic seq2seq task:
convert a number string like "142" to "one hundred forty two". The student
sees the architecture learn structure before facing real translation noise.
ONES = ["zero", "one", "two", "three", "four", "five", "six", "seven",
"eight", "nine", "ten", "eleven", "twelve", "thirteen", "fourteen",
"fifteen", "sixteen", "seventeen", "eighteen", "nineteen"]
TENS = ["", "", "twenty", "thirty", "forty", "fifty", "sixty", "seventy",
"eighty", "ninety"]
def number_to_words(n):
if n < 20:
return ONES[n]
if n < 100:
t, o = divmod(n, 10)
return TENS[t] + ((" " + ONES[o]) if o else "")
h, rem = divmod(n, 100)
# hundreds
words = ONES[h] + " hundred"
if rem:
words += " " + number_to_words(rem)
return words
def make_warmup_data(n_examples, seed=0):
rng = np.random.default_rng(seed)
nums = rng.integers(0, 1000, size=n_examples)
return [(str(int(n)), number_to_words(int(n))) for n in nums]
examples = make_warmup_data(8)
print("synthetic warmup pairs (source -> target):")
for s, t in examples:
print(" %4s -> %s" % (s, t))
if TORCH:
print()
print("[TORCH present] would train TinyTransformer on these until ~100% exact-match.")
else:
print()
print("[pre-computed] after warmup the model reaches ~99% exact-match in <60s CPU.")
8. Training note#
# Standard pattern for the real run on Multi30k:
#
# if TRAIN_FROM_SCRATCH:
# model = train(config) # ~10 min on a free Colab T4
# else:
# model = load_checkpoint(CKPT)
#
# python train.py --smoke # 500 pairs, 2 epochs, no eval -- used by CI
if TRAIN_FROM_SCRATCH and TORCH:
print("Training from scratch... (not run in this environment)")
else:
print("Loading checkpoint:", CKPT)
# pre-computed validation loss curve (cross-entropy) over 12 epochs
loss_curve = [6.21, 5.02, 4.18, 3.61, 3.20, 2.92, 2.71, 2.55, 2.43, 2.35, 2.30, 2.27]
print()
print("pre-computed validation loss curve (Multi30k EN->DE, tiny config):")
for ep, lv in enumerate(loss_curve, 1):
bar = "#" * int(round(lv * 6))
print("epoch %2d loss %.2f %s" % (ep, lv, bar))
print("final validation BLEU ~ 33.5")
9. Time-machine#
The tiny Transformer on the shared 20 held-out sentences. Output is clearly more fluent and accurate than the RNN-era systems – long-range agreement and word order are now handled well.
TIME_MACHINE_EN = [
"A man is walking his dog in the park .",
"Two children are playing near a fountain .",
"A woman in a red coat is reading a book .",
"Several people are waiting at a bus stop .",
"A cyclist is riding through a crowded street .",
"A dog is chasing a ball on the beach .",
"A group of tourists is taking photographs .",
"A young girl is feeding ducks by a pond .",
"Two men are playing chess in a cafe .",
"A street musician is playing the violin .",
"Children are running through a field of flowers .",
"An old man is sitting on a bench reading a newspaper .",
"A woman is carrying groceries up a flight of stairs .",
"A boy is kicking a football against a wall .",
"A couple is dancing on an empty street .",
"A chef is preparing food in an outdoor kitchen .",
"A cat is sleeping on a warm windowsill .",
"Workers are repairing a road in the rain .",
"A small boat is sailing on a calm lake .",
"Firefighters are climbing a tall ladder .",
]
# pre-computed tiny-Transformer EN->DE outputs (good quality)
TRANSFORMER_DE = [
"Ein Mann fuehrt seinen Hund im Park spazieren .",
"Zwei Kinder spielen in der Naehe eines Brunnens .",
"Eine Frau in einem roten Mantel liest ein Buch .",
"Mehrere Menschen warten an einer Bushaltestelle .",
"Ein Radfahrer faehrt durch eine belebte Strasse .",
"Ein Hund jagt einen Ball am Strand .",
"Eine Gruppe von Touristen macht Fotos .",
"Ein junges Maedchen fuettert Enten an einem Teich .",
"Zwei Maenner spielen Schach in einem Cafe .",
"Ein Strassenmusiker spielt Geige .",
"Kinder rennen durch ein Blumenfeld .",
"Ein alter Mann sitzt auf einer Bank und liest eine Zeitung .",
"Eine Frau traegt Lebensmittel eine Treppe hinauf .",
"Ein Junge schiesst einen Fussball gegen eine Wand .",
"Ein Paar tanzt auf einer leeren Strasse .",
"Ein Koch bereitet Essen in einer Aussenkueche zu .",
"Eine Katze schlaeft auf einem warmen Fensterbrett .",
"Arbeiter reparieren eine Strasse im Regen .",
"Ein kleines Boot segelt auf einem ruhigen See .",
"Feuerwehrleute klettern eine hohe Leiter hinauf .",
]
print("=== Time machine: Tiny Transformer (Vaswani 2017), EN -> DE ===")
for en, de in zip(TIME_MACHINE_EN, TRANSFORMER_DE):
print("EN:", en)
print("DE:", de)
print()
Interactive · Positional encodings, live#
Drag the dimension slider: low dimensions oscillate fast (fine-grained position), high dimensions oscillate slowly (coarse position). Together they give every position a unique fingerprint.