40 · The Neural Shift — Motivation & Reading

40 · The Neural Shift — Motivation & Reading#

Dependencies: torch (CPU)
Runtime: < 1 min

Chapter stub — implementation coming in a future commit.

Time-machine sentences#

Every chapter runs the same 20 held-out test sentences so the quality arc across chapters is directly comparable.

Open In Colab

# Colab setup: install this part's pinned dependencies (skipped outside Colab).
import sys
if "google.colab" in sys.modules:
    get_ipython().system("wget -q -O requirements_part4.txt https://raw.githubusercontent.com/eduardosanchezg/mthistory/main/requirements_part4.txt")
    get_ipython().system("pip install -q -r requirements_part4.txt")

The Neural Shift (2013-2014)#

Three papers in roughly eighteen months ended two decades of statistical machine translation (SMT) dominance and launched neural MT:

  • Kalchbrenner & Blunsom (2013) – Recurrent Continuous Translation Models (RCTM)

  • Cho et al. (2014) – RNN Encoder-Decoder (introduced the GRU)

  • Sutskever, Vinyals & Le (2014) – Sequence to Sequence Learning (LSTM seq2seq)

This chapter is mostly reading and motivation. We use tiny numpy toys to make the core ideas – and the core weakness (the fixed-vector bottleneck) – concrete. That weakness motivates attention, which is the next chapter.

1. Why SMT plateaued#

Phrase-based SMT (Part 3) was the state of the art for over a decade, but by 2013 it was hitting hard ceilings:

  • Feature-engineering ceiling. Every gain came from a hand-designed feature (lexical weights, reordering models, distortion penalties). Diminishing returns.

  • Phrase tables were enormous. Millions of entries, gigabytes on disk, yet still sparse: any unseen phrase fell back to word-by-word translation.

  • No generalization. SMT memorized surface phrases. walking his dog and walking her cat shared nothing – no notion that both are walking .

  • Brittle reordering. Long-distance reordering (e.g. German verb-final clauses) was handled by penalized search over permutations, which was both slow and fragile.

Neural models promised to replace all of this with learned continuous representations and a single end-to-end-trained network.

2. The three 2013-2014 papers#

Kalchbrenner & Blunsom (2013) – RCTM. A convolutional sentence model produced a fixed source representation; a recurrent model generated the target. First end-to-end continuous translation model. Crucially showed a single vector could condition generation of a whole sentence – but quality was modest and it was hard to train.

Cho et al. (2014) – RNN Encoder-Decoder. Two RNNs: an encoder reads the source into a fixed-length vector c; a decoder generates the target conditioned on c. Introduced the GRU (Gated Recurrent Unit) to fight vanishing gradients. Originally used to rescore phrase-based SMT candidates, not translate alone.

Sutskever et al. (2014) – seq2seq. Deep LSTM encoder-decoder that translated on its own, competitively with SMT on WMT’14 EN-FR. Two famous tricks:

  1. Reverse the source sentence. Feeding the source backwards put the first source words close (in time steps) to the first target words, shortening dependencies and dramatically improving optimization.

  2. Deep stacked LSTMs + large beam search.

The shared idea across all three: compress the source sentence into a fixed vector, then decode the target from it. The next sections show why that compression is the Achilles’ heel.

3. The encoder-decoder idea (numpy toy)#

Encode a 10-word sentence into a single 4-dimensional vector, then ask: how much did we lose? We use random embeddings + a random linear ‘encoder’ and measure reconstruction error. A small bottleneck cannot hold everything.

import numpy as np
rng = np.random.default_rng(0)

EMB_DIM = 16        # per-word embedding size
SENT_LEN = 10       # words in the sentence
BOTTLENECK = 4      # fixed vector size c

# Random "sentence": 10 word vectors of dim 16.
sentence = rng.standard_normal((SENT_LEN, EMB_DIM))

# Encoder: flatten the sentence and project down to a fixed 4-dim vector c.
flat = sentence.reshape(-1)                 # 160-dim
W_enc = rng.standard_normal((BOTTLENECK, flat.size)) / np.sqrt(flat.size)
c = np.tanh(W_enc @ flat)                   # the fixed-length context vector

# Decoder: try to reconstruct the original 160-dim sentence from just c.
W_dec = rng.standard_normal((flat.size, BOTTLENECK)) / np.sqrt(BOTTLENECK)
recon = W_dec @ c

err = np.mean((recon - flat) ** 2)
var = np.var(flat)
print("Sentence  :", SENT_LEN, "words x", EMB_DIM, "dims =", flat.size, "numbers")
print("Bottleneck:", BOTTLENECK, "numbers (the fixed context vector c)")
print("Compression ratio: {:.0f}x".format(flat.size / BOTTLENECK))
print()
print("Reconstruction MSE      : {:.3f}".format(err))
print("Variance of the signal  : {:.3f}".format(var))
print("Fraction of info lost   : {:.0%}".format(min(1.0, err / var)))
print()
print("Squeezing 160 numbers through 4 is hugely lossy. Real encoders learn")
print("much better compressions -- but the *fixed size* is the structural limit.")

4. The fixed-vector bottleneck vs. sentence length#

Cho et al. (2014) reported that fixed-vector encoder-decoders degrade sharply as sentences get longer: a single vector simply cannot hold a 40-word sentence as well as a 7-word one. Below we simulate this with a numpy toy – as we push more words through the same fixed bottleneck, reconstruction error rises. The illustrative ‘BLEU’ numbers mirror the curve from the Cho et al. paper (Figure 2).

import numpy as np
rng = np.random.default_rng(1)

EMB_DIM = 16
BOTTLENECK = 4
lengths = [5, 10, 15, 20, 30, 40, 50]

# Pre-computed illustrative BLEU for a *fixed-vector* model (drops with length),
# mirroring Cho et al. 2014 Fig. 2. A model WITH attention would stay roughly flat.
bleu_fixed = [28.5, 28.0, 26.5, 24.0, 19.5, 14.0, 9.5]
bleu_attn  = [28.6, 28.4, 28.1, 27.6, 27.0, 26.2, 25.4]

print("Toy bottleneck reconstruction error vs. sentence length")
print("len   recon_MSE   (more words -> same 4-dim vector -> worse)")
for L in lengths:
    flat = rng.standard_normal(L * EMB_DIM)
    W_enc = rng.standard_normal((BOTTLENECK, flat.size)) / np.sqrt(flat.size)
    c = np.tanh(W_enc @ flat)
    W_dec = rng.standard_normal((flat.size, BOTTLENECK)) / np.sqrt(BOTTLENECK)
    err = np.mean((W_dec @ c - flat) ** 2)
    print("{:>3}   {:>8.3f}".format(L, err))

print()
print("Illustrative BLEU by source length (cf. Cho et al. 2014):")
print("len | fixed-vector | with-attention")
maxb = max(bleu_attn)
for L, bf, ba in zip(lengths, bleu_fixed, bleu_attn):
    bar_f = "#" * int(round(bf / maxb * 30))
    print("{:>3} | {:>5.1f} {:<31} {:>5.1f}".format(L, bf, bar_f, ba))
print()
print("The fixed-vector curve collapses past ~20 words; attention (next chapter)")
print("stays nearly flat because it can re-read any source word on demand.")

5. Time-machine: 2014-era neural MT#

Pre-computed outputs from an early fixed-vector RNN encoder-decoder. Notice the 2014 characteristics: mostly fluent short sentences, occasional wrong word choice or dropped detail, and visible trouble on the longer sentences (the bottleneck at work).

# Time-machine: 2014-era fixed-vector RNN encoder-decoder
# Decent on short sentences; drops detail on longer ones (the bottleneck).
TIME_MACHINE_EN = [
    "A man is walking his dog in the park .",
    "Two children are playing near a fountain .",
    "A woman in a red coat is reading a book .",
    "Several people are waiting at a bus stop .",
    "A cyclist is riding through a crowded street .",
    "A dog is chasing a ball on the beach .",
    "A group of tourists is taking photographs .",
    "A young girl is feeding ducks by a pond .",
    "Two men are playing chess in a cafe .",
    "A street musician is playing the violin .",
    "Children are running through a field of flowers .",
    "An old man is sitting on a bench reading a newspaper .",
    "A woman is carrying groceries up a flight of stairs .",
    "A boy is kicking a football against a wall .",
    "A couple is dancing on an empty street .",
    "A chef is preparing food in an outdoor kitchen .",
    "A cat is sleeping on a warm windowsill .",
    "Workers are repairing a road in the rain .",
    "A small boat is sailing on a calm lake .",
    "Firefighters are climbing a tall ladder .",
]
TIME_MACHINE_DE = [
    "Ein Mann geht mit seinem Hund im Park spazieren .",
    "Zwei Kinder spielen neben einem Brunnen .",
    "Eine Frau in einem roten Mantel liest ein Buch .",
    "Mehrere Menschen warten an einer Bushaltestelle .",
    "Ein Radfahrer faehrt durch eine belebte Strasse .",
    "Ein Hund jagt einen Ball am Strand .",
    "Eine Gruppe von Touristen macht Fotos .",
    "Ein junges Maedchen fuettert Enten an einem Teich .",
    "Zwei Maenner spielen Schach in einem Cafe .",
    "Ein Strassenmusiker spielt die Geige .",
    "Kinder laufen durch ein Feld von Blumen .",
    "Ein alter Mann sitzt auf einer Bank und liest eine Zeitung .",
    "Eine Frau traegt Lebensmittel eine Treppe .",
    "Ein Junge tritt einen Fussball gegen eine Wand .",
    "Ein Paar tanzt auf einer leeren Strasse .",
    "Ein Koch bereitet Essen in einer Kueche zu .",
    "Eine Katze schlaeft auf einem warmen Fenster .",
    "Arbeiter reparieren eine Strasse im Regen .",
    "Ein kleines Boot segelt auf einem ruhigen See .",
    "Feuerwehrleute klettern eine hohe Leiter .",
]

print("=" * 70)
print("2014-era fixed-vector RNN encoder-decoder")
print("=" * 70)
for en, de in zip(TIME_MACHINE_EN, TIME_MACHINE_DE):
    print("EN:", en)
    print("DE:", de)
    print("-" * 70)
Contents