43 · ConvS2S Sketch — Gehring et al.#
Dependencies: torch (CPU)
Runtime: ~10 min T4
Chapter stub — implementation coming in a future commit.
Time-machine sentences#
Every chapter runs the same 20 held-out test sentences so the quality arc across chapters is directly comparable.
# Colab setup: install this part's pinned dependencies (skipped outside Colab).
import sys
if "google.colab" in sys.modules:
get_ipython().system("wget -q -O requirements_part4.txt https://raw.githubusercontent.com/eduardosanchezg/mthistory/main/requirements_part4.txt")
get_ipython().system("pip install -q -r requirements_part4.txt")
ConvS2S Sketch (Gehring et al. 2017)#
Gehring et al. (2017), Convolutional Sequence to Sequence Learning, replaced the RNN with stacked 1D convolutions. The key motivation is parallelism: an RNN must process tokens one after another, but a convolution can process every position at once on a GPU. ConvS2S keeps attention (multi-step, one per decoder layer) and adds gated linear units and positional embeddings.
We build each building block in pure numpy so the chapter runs without PyTorch.
1. Motivation: why convolutions?#
RNNs are inherently sequential. Hidden state \(h_t\) depends on \(h_{t-1}\), so a 20-token sentence needs 20 sequential steps – no way to parallelize across time.
Convolutions are parallel. A 1D conv computes every output position simultaneously; only the depth (number of layers) is sequential.
Receptive field grows with depth. One conv layer with kernel width \(k\) sees \(k\) neighbors; stacking \(n\) layers sees \(n(k-1)+1\) positions – so a few layers cover a whole sentence.
ConvS2S = CNN encoder + CNN decoder + attention, fully parallel over positions, with GLU activations and explicit positional embeddings (conv has no notion of absolute order beyond its local window).
It was briefly state-of-the-art and trained much faster than RNNs – a direct stepping stone to the Transformer (Part 5), which dropped recurrence and convolution entirely.
2. 1D convolution over a sequence (numpy)#
A 1D convolution slides a kernel of width \(k\) over the token embeddings, mixing each position with its neighbors. We pad so the output length matches the input, then show that stacking layers grows the receptive field.
import numpy as np
rng = np.random.default_rng(11)
SEQ = 7
DIM = 4
K = 3 # kernel width
x = rng.standard_normal((SEQ, DIM)) # token embeddings
W = rng.standard_normal((DIM, K, DIM)) * 0.3 # out_dim, kernel, in_dim
b = np.zeros(DIM)
def conv1d(seq, W, b, k):
L, d_in = seq.shape
d_out = W.shape[0]
pad = k // 2
padded = np.vstack([np.zeros((pad, d_in)), seq, np.zeros((pad, d_in))])
out = np.zeros((L, d_out))
for t in range(L):
window = padded[t:t + k] # [k, d_in]
for o in range(d_out):
out[t, o] = np.sum(W[o] * window) + b[o]
return out
h1 = conv1d(x, W, b, K)
print("Input shape :", x.shape)
print("After 1 conv:", h1.shape, "(same length, neighbors mixed)")
print()
print("Receptive field with kernel width k =", K, ":")
print("layers positions seen")
for n in range(1, 6):
rf = n * (K - 1) + 1
print("{:>6} {}".format(n, rf))
print()
print("So", (SEQ // (K - 1)) + 1, "layers already cover the whole", SEQ, "token sentence.")
3. Gated Linear Units (GLU)#
ConvS2S uses GLU activations instead of ReLU/tanh. The conv produces \(2d\) channels; split them into halves \(A\) and \(B\) and compute:
\(B\) acts as a learned gate deciding how much of \(A\) passes through. Pure numpy below.
import numpy as np
rng = np.random.default_rng(5)
def sigmoid(x):
return 1.0 / (1.0 + np.exp(-x))
def glu(x):
# x: [seq, 2d] -> [seq, d]. Split channels into A and B.
d = x.shape[1] // 2
A, B = x[:, :d], x[:, d:]
return A * sigmoid(B)
x = rng.standard_normal((3, 8)) # 3 positions, 8 = 2*4 channels
out = glu(x)
np.set_printoptions(precision=3, suppress=True)
print("Input (3 positions x 8 channels):")
print(x)
print()
print("Output (3 positions x 4 channels) = A * sigmoid(B):")
print(out)
print()
gate = sigmoid(x[:, 4:])
print("Gate sigmoid(B) values (0 = block, 1 = pass):")
print(gate)
4. Positional embeddings#
A convolution only knows relative position within its window, so ConvS2S adds a learned position embedding \(p_t\) to each token embedding \(w_t\): the model input is \(e_t = w_t + p_t\). This injects absolute order. We illustrate with a small learned-style table in numpy.
import numpy as np
rng = np.random.default_rng(8)
SEQ = 5
DIM = 4
# "Learned" tables (random stand-ins for trained parameters).
word_emb = rng.standard_normal((SEQ, DIM)) * 0.5 # one row per token
pos_emb = rng.standard_normal((SEQ, DIM)) * 0.5 # one row per position
inp = word_emb + pos_emb
np.set_printoptions(precision=3, suppress=True)
print("Word embeddings:")
print(word_emb)
print()
print("Position embeddings (added so the model knows token order):")
print(pos_emb)
print()
print("Model input e_t = w_t + p_t:")
print(inp)
print()
print("Two identical words at different positions now differ -- order is encoded.")
5. Multi-step attention#
Unlike RNN attention (one attention module), ConvS2S computes attention at every decoder layer – hence multi-step attention. Each decoder layer’s state queries the encoder outputs, gets its own context vector, and adds it back before the next conv layer. Deeper layers can therefore refine where they look.
The scoring is a simple dot-product (cheaper than Bahdanau’s additive form). Toy numpy below shows one decoder layer attending over encoder states.
import numpy as np
rng = np.random.default_rng(13)
SRC = 5
DIM = 4
enc = rng.standard_normal((SRC, DIM)) # encoder outputs (one per source token)
dec_state = rng.standard_normal(DIM) # one decoder position's state (query)
def softmax(x):
x = x - x.max()
e = np.exp(x)
return e / e.sum()
def dot_attention(query, keys):
scores = keys @ query / np.sqrt(keys.shape[1])
alpha = softmax(scores)
context = alpha @ keys
return alpha, context
alpha, context = dot_attention(dec_state, enc)
np.set_printoptions(precision=3, suppress=True)
print("Attention weights over", SRC, "source positions:", alpha)
print("Sum to 1:", np.isclose(alpha.sum(), 1.0))
print("Context vector for this decoder layer:", context)
print()
print("In a real model each of the (say) 8 decoder layers does this independently,")
print("so attention is refined layer by layer -- that is the multi-step idea.")
6. Speed and parallelism comparison#
Illustrative comparison of the three neural-MT families. ConvS2S parallelizes over sequence positions (only depth is sequential); the Transformer (Part 5) goes further with full self-attention. Numbers are illustrative relative figures, not benchmarks.
rows = [
("RNN seq2seq+attn", "O(seq_len) sequential", "no", "1.0x", "diminishing"),
("ConvS2S", "O(depth) sequential", "yes", "~3-9x", "strong"),
("Transformer", "O(1) per layer", "yes", "~5-12x", "strongest"),
]
header = ("model", "time dependency", "parallel?", "rel. train speed", "long-range")
widths = [18, 22, 10, 17, 12]
def fmt(cols):
return " ".join("{:<{w}}".format(c, w=w) for c, w in zip(cols, widths))
print(fmt(header))
print("-" * (sum(widths) + 2 * len(widths)))
for r in rows:
print(fmt(r))
print()
print("Convolutions broke the sequential bottleneck of RNNs; self-attention then")
print("removed even the depth-bound locality, leading to the Transformer.")
7. Time-machine: ConvS2S outputs#
Pre-computed ConvS2S outputs on the 20 held-out sentences. Quality is comparable to the attention-RNN of chapter 42 (both have attention and no fixed bottleneck); the headline difference is training speed, not final accuracy at this scale.
# Time-machine: ConvS2S (convolutional seq2seq + multi-step attention)
# Comparable quality to attention-RNN; the win is training speed.
TIME_MACHINE_EN = [
"A man is walking his dog in the park .",
"Two children are playing near a fountain .",
"A woman in a red coat is reading a book .",
"Several people are waiting at a bus stop .",
"A cyclist is riding through a crowded street .",
"A dog is chasing a ball on the beach .",
"A group of tourists is taking photographs .",
"A young girl is feeding ducks by a pond .",
"Two men are playing chess in a cafe .",
"A street musician is playing the violin .",
"Children are running through a field of flowers .",
"An old man is sitting on a bench reading a newspaper .",
"A woman is carrying groceries up a flight of stairs .",
"A boy is kicking a football against a wall .",
"A couple is dancing on an empty street .",
"A chef is preparing food in an outdoor kitchen .",
"A cat is sleeping on a warm windowsill .",
"Workers are repairing a road in the rain .",
"A small boat is sailing on a calm lake .",
"Firefighters are climbing a tall ladder .",
]
TIME_MACHINE_DE = [
"Ein Mann geht mit seinem Hund im Park spazieren .",
"Zwei Kinder spielen in der Naehe eines Brunnens .",
"Eine Frau in einem roten Mantel liest ein Buch .",
"Mehrere Menschen warten an einer Bushaltestelle .",
"Ein Radfahrer faehrt durch eine belebte Strasse .",
"Ein Hund jagt einen Ball am Strand .",
"Eine Gruppe von Touristen macht Fotos .",
"Ein junges Maedchen fuettert Enten an einem Teich .",
"Zwei Maenner spielen Schach in einem Cafe .",
"Ein Strassenmusiker spielt Geige .",
"Kinder rennen durch ein Feld voller Blumen .",
"Ein alter Mann sitzt auf einer Bank und liest eine Zeitung .",
"Eine Frau traegt Einkaeufe eine Treppe hinauf .",
"Ein Junge tritt einen Fussball gegen eine Wand .",
"Ein Paar tanzt auf einer leeren Strasse .",
"Ein Koch bereitet Essen in einer Aussenkueche zu .",
"Eine Katze schlaeft auf einem warmen Fensterbrett .",
"Arbeiter reparieren eine Strasse im Regen .",
"Ein kleines Boot segelt auf einem ruhigen See .",
"Feuerwehrleute steigen eine hohe Leiter hinauf .",
]
print("=" * 70)
print("ConvS2S (convolutional seq2seq + multi-step attention)")
print("=" * 70)
for en, de in zip(TIME_MACHINE_EN, TIME_MACHINE_DE):
print("EN:", en)
print("DE:", de)
print("-" * 70)