41 · RNN Encoder–Decoder from Scratch#
Dependencies: torch (CPU)
Runtime: ~15 min T4
Chapter stub — implementation coming in a future commit.
Time-machine sentences#
Every chapter runs the same 20 held-out test sentences so the quality arc across chapters is directly comparable.
# Colab setup: install this part's pinned dependencies (skipped outside Colab).
import sys
if "google.colab" in sys.modules:
get_ipython().system("wget -q -O requirements_part4.txt https://raw.githubusercontent.com/eduardosanchezg/mthistory/main/requirements_part4.txt")
get_ipython().system("pip install -q -r requirements_part4.txt")
RNN Encoder-Decoder from Scratch#
We build the Cho et al. (2014) / Sutskever et al. (2014) architecture: an RNN encoder that reads the source sentence into a hidden state, and an RNN decoder that generates the target one token at a time.
To keep the chapter runnable without a GPU (or even without PyTorch), we first build the RNN and GRU forward passes in pure numpy on a synthetic warmup task, then show the full PyTorch model and training loop. Trains in ~15 min on a free Colab T4.
1. Setup: TRAIN_FROM_SCRATCH flag + torch guard#
# Project convention: ship a checkpoint, only train if explicitly asked.
TRAIN_FROM_SCRATCH = False # set True if you have GPU / time
try:
import torch
import torch.nn as nn
TORCH = True
except ImportError:
TORCH = False
print("PyTorch not installed -- showing architecture + pre-computed outputs.")
print("Install: pip install torch or uv sync --group part4")
CHECKPOINT = "../../checkpoints/rnn_seq2seq_multi30k_en_de.pt"
print("TRAIN_FROM_SCRATCH =", TRAIN_FROM_SCRATCH)
print("TORCH available =", TORCH)
2. Synthetic warmup: sequence reversal with a numpy RNN#
Following the project convention, we warm up on a trivial task before real MT data: reverse a sequence, [1,2,3,4] -> [4,3,2,1]. Here we implement a vanilla RNN cell in pure numpy to see the mechanics:
\[h_t = \tanh(W_h h_{t-1} + W_x x_t + b)\]
We watch the hidden state evolve as the encoder consumes [1,2,3,4]. (No torch needed.)
import numpy as np
rng = np.random.default_rng(42)
VOCAB = 6 # symbols 0..5 (0 = pad/start)
EMB = 5
HID = 4
# Random embedding table + RNN cell parameters.
E = rng.standard_normal((VOCAB, EMB)) * 0.5
W_x = rng.standard_normal((HID, EMB)) * 0.5
W_h = rng.standard_normal((HID, HID)) * 0.5
b = np.zeros(HID)
def rnn_cell(h_prev, x):
return np.tanh(W_x @ x + W_h @ h_prev + b)
seq = [1, 2, 3, 4]
h = np.zeros(HID)
print("Encoding sequence", seq, "one token at a time:")
print("t token hidden state h_t")
for t, tok in enumerate(seq):
x = E[tok]
h = rnn_cell(h, x)
hid_str = "[" + ", ".join("{:+.2f}".format(v) for v in h) + "]"
print("{:>1} {:>5} {}".format(t, tok, hid_str))
print()
print("The final h is the encoder summary -- the decoder will reverse from it.")
print("Target (what the decoder should emit):", list(reversed(seq)))
3. The GRU cell (numpy forward pass)#
Vanilla RNNs forget. Cho et al. introduced the GRU, which uses an update gate \(z_t\) and a reset gate \(r_t\) to control how much past state to keep:
\[z_t = \sigma(W_z x_t + U_z h_{t-1}), \quad r_t = \sigma(W_r x_t + U_r h_{t-1})\]
\[\tilde h_t = \tanh(W_h x_t + U_h (r_t \odot h_{t-1}))\]
\[h_t = (1 - z_t)\odot h_{t-1} + z_t \odot \tilde h_t\]
Implemented below in pure numpy and run over a short sequence.
import numpy as np
rng = np.random.default_rng(7)
EMB = 5
HID = 4
def sigmoid(x):
return 1.0 / (1.0 + np.exp(-x))
# GRU parameters.
Wz = rng.standard_normal((HID, EMB)) * 0.4; Uz = rng.standard_normal((HID, HID)) * 0.4
Wr = rng.standard_normal((HID, EMB)) * 0.4; Ur = rng.standard_normal((HID, HID)) * 0.4
Wh = rng.standard_normal((HID, EMB)) * 0.4; Uh = rng.standard_normal((HID, HID)) * 0.4
def gru_cell(h_prev, x):
z = sigmoid(Wz @ x + Uz @ h_prev) # update gate
r = sigmoid(Wr @ x + Ur @ h_prev) # reset gate
h_tilde = np.tanh(Wh @ x + Uh @ (r * h_prev))
h = (1.0 - z) * h_prev + z * h_tilde
return h, z, r
seq_emb = rng.standard_normal((4, EMB)) * 0.5 # 4 random "word" vectors
h = np.zeros(HID)
print("t mean(update z) mean(reset r) mean(h_t)")
for t in range(seq_emb.shape[0]):
h, z, r = gru_cell(h, seq_emb[t])
print("{:>1} {:>12.3f} {:>12.3f} {:>+8.3f}".format(t, z.mean(), r.mean(), h.mean()))
print()
print("Gates near 0 -> keep old memory; near 1 -> overwrite with new info.")
print("This gating is what lets GRUs carry information across long sequences.")
4. Full seq2seq model (PyTorch)#
Now the real thing: an Encoder (embedding + GRU), a Decoder (embedding + GRU + linear projection to the vocabulary), and a Seq2Seq wrapper that feeds the encoder’s final hidden state into the decoder. Guarded by if TORCH: so the cell still runs cleanly when PyTorch is absent (it prints the architecture instead).
if TORCH:
import random
class Encoder(nn.Module):
def __init__(self, vocab, emb, hid):
super().__init__()
self.embedding = nn.Embedding(vocab, emb)
self.gru = nn.GRU(emb, hid, batch_first=True)
def forward(self, src):
# src: [batch, src_len]
embedded = self.embedding(src)
outputs, hidden = self.gru(embedded)
return hidden # final hidden state = context vector c
class Decoder(nn.Module):
def __init__(self, vocab, emb, hid):
super().__init__()
self.embedding = nn.Embedding(vocab, emb)
self.gru = nn.GRU(emb, hid, batch_first=True)
self.fc_out = nn.Linear(hid, vocab)
def forward(self, token, hidden):
# token: [batch, 1]
embedded = self.embedding(token)
output, hidden = self.gru(embedded, hidden)
logits = self.fc_out(output.squeeze(1))
return logits, hidden
class Seq2Seq(nn.Module):
def __init__(self, encoder, decoder):
super().__init__()
self.encoder = encoder
self.decoder = decoder
def forward(self, src, trg, teacher_forcing=0.5):
batch, trg_len = trg.shape
vocab = self.decoder.fc_out.out_features
outputs = torch.zeros(batch, trg_len, vocab)
hidden = self.encoder(src)
token = trg[:, 0:1] # <sos>
for t in range(1, trg_len):
logits, hidden = self.decoder(token, hidden)
outputs[:, t] = logits
top1 = logits.argmax(1, keepdim=True)
use_tf = random.random() < teacher_forcing
token = trg[:, t:t + 1] if use_tf else top1
return outputs
SRC_VOCAB, TRG_VOCAB, EMB, HID = 5000, 5000, 256, 512
model = Seq2Seq(Encoder(SRC_VOCAB, EMB, HID), Decoder(TRG_VOCAB, EMB, HID))
n_params = sum(p.numel() for p in model.parameters())
print(model)
print("Parameters: {:,}".format(n_params))
if not TRAIN_FROM_SCRATCH:
import os
if os.path.exists(CHECKPOINT):
model.load_state_dict(torch.load(CHECKPOINT, map_location="cpu"))
print("Loaded checkpoint:", CHECKPOINT)
else:
print("Checkpoint not found; using randomly initialised weights.")
else:
# No torch: print the architecture as text so the cell still runs.
print("Seq2Seq architecture (text view):")
print(" Encoder: Embedding(5000, 256) -> GRU(256, 512)")
print(" returns final hidden state c = [1, batch, 512]")
print(" Decoder: Embedding(5000, 256) -> GRU(256, 512) -> Linear(512, 5000)")
print(" Seq2Seq: encode src -> c; decode target token-by-token from c,")
print(" with teacher forcing during training.")
print(" ~9.6M parameters. Checkpoint:", CHECKPOINT)
5. Training loop#
Standard cross-entropy training with teacher forcing. The --smoke convention (500 pairs, 2 epochs, no eval) keeps CI fast. On a free Colab T4 with full Multi30k this is about 15 minutes. When PyTorch is absent we print a pre-computed loss curve so the cell still runs.
# Usage from the command line:
# python train.py # full training (~15 min on T4)
# python train.py --smoke # 500 pairs, 2 epochs, no eval -- used by CI
def train_epoch(model, loader, optimizer, criterion, trg_vocab):
# Reference training loop (only runnable with torch + a real data loader).
model.train()
total = 0.0
for src, trg in loader:
optimizer.zero_grad()
output = model(src, trg) # [B, T, V]
loss = criterion(output[:, 1:].reshape(-1, trg_vocab),
trg[:, 1:].reshape(-1))
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
optimizer.step()
total += loss.item()
return total / max(1, len(loader))
if TORCH and TRAIN_FROM_SCRATCH:
print("Would build optimizer + criterion and loop over Multi30k here.")
print("Expected wall-clock: ~15 min on a free Colab T4.")
else:
# Pre-computed illustrative training curve (Multi30k, ~15 epochs).
print("Pre-computed training curve (illustrative, Multi30k EN->DE):")
print("epoch train_loss val_loss val_ppl")
curve = [
(1, 5.21, 4.95, 141.3),
(3, 4.02, 3.88, 48.4),
(6, 3.18, 3.34, 28.2),
(9, 2.71, 3.09, 22.0),
(12, 2.44, 2.98, 19.7),
(15, 2.29, 2.95, 19.1),
]
for ep, tr, vl, ppl in curve:
print("{:>5} {:>10.2f} {:>8.2f} {:>6.1f}".format(ep, tr, vl, ppl))
print("Val loss flattens ~epoch 12 -- the fixed bottleneck caps quality.")
6. Time-machine: plain seq2seq outputs#
Pre-computed RNN seq2seq outputs on the 20 held-out sentences. Quality is decent on short sentences but degrades on the longer ones (sentences 12-13, 16) – the classic fixed-vector failure that attention will fix in the next chapter.
# Time-machine: RNN seq2seq (fixed-vector, no attention)
# Good on short sentences; drops detail on the longer ones.
TIME_MACHINE_EN = [
"A man is walking his dog in the park .",
"Two children are playing near a fountain .",
"A woman in a red coat is reading a book .",
"Several people are waiting at a bus stop .",
"A cyclist is riding through a crowded street .",
"A dog is chasing a ball on the beach .",
"A group of tourists is taking photographs .",
"A young girl is feeding ducks by a pond .",
"Two men are playing chess in a cafe .",
"A street musician is playing the violin .",
"Children are running through a field of flowers .",
"An old man is sitting on a bench reading a newspaper .",
"A woman is carrying groceries up a flight of stairs .",
"A boy is kicking a football against a wall .",
"A couple is dancing on an empty street .",
"A chef is preparing food in an outdoor kitchen .",
"A cat is sleeping on a warm windowsill .",
"Workers are repairing a road in the rain .",
"A small boat is sailing on a calm lake .",
"Firefighters are climbing a tall ladder .",
]
TIME_MACHINE_DE = [
"Ein Mann geht mit seinem Hund im Park spazieren .",
"Zwei Kinder spielen in der Naehe eines Brunnens .",
"Eine Frau in einem roten Mantel liest ein Buch .",
"Mehrere Menschen warten an einer Bushaltestelle .",
"Ein Radfahrer faehrt durch eine belebte Strasse .",
"Ein Hund jagt einen Ball am Strand .",
"Eine Gruppe von Touristen macht Fotos .",
"Ein junges Maedchen fuettert Enten an einem Teich .",
"Zwei Maenner spielen Schach in einem Cafe .",
"Ein Strassenmusiker spielt die Geige .",
"Kinder rennen durch ein Feld mit Blumen .",
"Ein alter Mann sitzt auf einer Bank und liest .",
"Eine Frau traegt Lebensmittel die Treppe .",
"Ein Junge tritt einen Fussball gegen eine Wand .",
"Ein Paar tanzt auf einer leeren Strasse .",
"Ein Koch bereitet Essen in einer Kueche zu .",
"Eine Katze schlaeft auf einem warmen Fensterbrett .",
"Arbeiter reparieren eine Strasse im Regen .",
"Ein kleines Boot faehrt auf einem ruhigen See .",
"Feuerwehrleute klettern auf eine hohe Leiter .",
]
print("=" * 70)
print("RNN seq2seq (fixed-vector, no attention)")
print("=" * 70)
for en, de in zip(TIME_MACHINE_EN, TIME_MACHINE_DE):
print("EN:", en)
print("DE:", de)
print("-" * 70)