52 · Scaling — WMT, Back-Translation, Filtering#
Dependencies: torch
Runtime: reference only
Chapter stub — implementation coming in a future commit.
Time-machine sentences#
Every chapter runs the same 20 held-out test sentences so the quality arc across chapters is directly comparable.
# Colab setup: install this part's pinned dependencies (skipped outside Colab).
import sys
if "google.colab" in sys.modules:
get_ipython().system("wget -q -O requirements_part5.txt https://raw.githubusercontent.com/eduardosanchezg/mthistory/main/requirements_part5.txt")
get_ipython().system("pip install -q -r requirements_part5.txt")
Scaling: WMT Systems, Back-Translation, Data Filtering#
Once the Transformer architecture was fixed, progress came from scale and data engineering rather than new model designs. This reference chapter covers the three pillars of modern WMT-winning systems: leveraging monolingual data with back-translation, cleaning parallel corpora with filtering, and combining models via ensembling. The toy implementations run with stdlib only.
2. Back-translation#
Parallel data is scarce; monolingual target-language text is abundant. Sennrich et al. (2016) showed you can:
Train a reverse model (target -> source).
Run it on monolingual target sentences to produce synthetic source sentences.
Add these (synthetic-source, real-target) pairs to your training data.
The synthetic source side is noisy, but the target side is real fluent text, which is exactly what the decoder needs to learn. Below we simulate the reverse model with a trivial word-substitution dictionary.
# toy "reverse model": German monolingual -> synthetic English (word substitution)
REV_DICT = {
"ein": "a", "eine": "a", "mann": "man", "frau": "woman", "hund": "dog",
"katze": "cat", "kinder": "children", "spielen": "play", "liest": "reads",
"buch": "book", "im": "in the", "park": "park", "strasse": "street",
"laeuft": "runs", "sitzt": "sits", "auf": "on", "der": "the", "die": "the",
}
def reverse_fn(target_sentence):
# simulate a target->source translation model
out = []
for tok in target_sentence.lower().split():
out.append(REV_DICT.get(tok, tok)) # unknown words pass through (noisy)
return " ".join(out)
def back_translate(mono, reverse_fn):
# mono: list of monolingual TARGET-side sentences
# returns synthetic (source, target) training pairs
pairs = []
for tgt in mono:
synthetic_src = reverse_fn(tgt)
pairs.append((synthetic_src, tgt))
return pairs
mono_de = [
"Ein Mann liest ein Buch",
"Die Kinder spielen im Park",
"Eine Katze sitzt auf der Strasse",
]
real_parallel = [
("a dog runs", "Ein Hund laeuft"),
("a woman reads", "Eine Frau liest"),
]
synthetic = back_translate(mono_de, reverse_fn)
print("synthetic (back-translated) pairs:")
for s, t in synthetic:
print(" SRC(synthetic):", s)
print(" TGT(real) :", t)
print()
augmented = real_parallel + synthetic
print("real parallel pairs: ", len(real_parallel))
print("after adding BT data: ", len(augmented))
print("effective corpus growth: %.1fx" % (len(augmented) / len(real_parallel)))
3. Data filtering#
Web-crawled parallel data (e.g. ParaCrawl) is noisy: misaligned pairs, boilerplate, wrong languages, duplicates. A few cheap heuristics remove most of the garbage:
length-ratio filter: drop pairs where
len(src)/len(tgt)is extreme (likely misaligned).length filter: drop empty / absurdly long sentences.
dedup: drop exact duplicate pairs.
noisy_corpus = [
("a dog runs", "Ein Hund laeuft"),
("a woman reads a book", "Eine Frau liest ein Buch"),
("hello", "Dies ist ein viel zu langer Satz der nicht zur Quelle passt ueberhaupt nicht"),
("", "Leerer Quellsatz"),
("a dog runs", "Ein Hund laeuft"),
("this is a very long english sentence with many many words here", "kurz"),
("two children play", "Zwei Kinder spielen"),
]
def length_ratio_ok(src, tgt, max_ratio=2.5):
ls, lt = len(src.split()), len(tgt.split())
if ls == 0 or lt == 0:
return False
ratio = max(ls, lt) / min(ls, lt)
return ratio <= max_ratio
def length_ok(src, tgt, min_len=1, max_len=40):
ls, lt = len(src.split()), len(tgt.split())
return min_len <= ls <= max_len and min_len <= lt <= max_len
def filter_corpus(corpus):
seen = set()
kept, dropped = [], []
for src, tgt in corpus:
key = (src, tgt)
if key in seen:
dropped.append((src, tgt, "duplicate"))
continue
if not length_ok(src, tgt):
dropped.append((src, tgt, "length"))
continue
if not length_ratio_ok(src, tgt):
dropped.append((src, tgt, "length-ratio"))
continue
seen.add(key)
kept.append((src, tgt))
return kept, dropped
kept, dropped = filter_corpus(noisy_corpus)
print("input pairs: ", len(noisy_corpus))
print("kept pairs: ", len(kept))
print("dropped: ", len(dropped))
print()
print("dropped reasons:")
for src, tgt, why in dropped:
shown = (src[:30] + "...") if len(src) > 30 else src
print(" [%-12s] %r" % (why, shown))
4. Scaling laws#
Translation quality grows roughly logarithmically with parallel data size: each doubling buys a smaller-but-steady BLEU gain. Below are illustrative (pre-computed) numbers for a fixed Transformer on increasing EN->DE data.
import math
# (sentence pairs, illustrative BLEU)
scaling = [
(29_000, 24.0),
(100_000, 28.5),
(500_000, 32.0),
(2_000_000, 35.5),
(10_000_000, 38.5),
(40_000_000, 40.5),
]
print("%-14s %-8s %s" % ("pairs", "BLEU", "curve"))
print("-" * 52)
for n, bleu in scaling:
bar = "#" * int(round(bleu))
print("%-14s %-8.1f %s" % ("{:,}".format(n), bleu, bar))
print()
print("note the diminishing returns: ~4.5 BLEU from 29k->100k,")
print("but only ~2 BLEU from 10M->40M (roughly linear in log(data)).")
5. Ensembling and checkpoint averaging#
Two cheap tricks that reliably add 0.5-1.5 BLEU at no training cost:
Ensembling: run several independently-trained models and average their output probability distributions at each decoding step. More diverse models -> bigger gain, but inference cost scales with the number of models.
Checkpoint averaging: average the weights of the last N checkpoints from a single training run. This costs nothing extra at inference (one model) and smooths out the noise of the final SGD steps. The Transformer paper averaged the last 5-20 checkpoints.
Most WMT-winning submissions combine both: ensemble several checkpoint-averaged models.
6. Time-machine#
A strong, scaled WMT-class system on the shared 20 sentences. With back-translation, big data, and ensembling, output is near-human fluency. Always runs (pre-computed).
TIME_MACHINE_EN = [
"A man is walking his dog in the park .",
"Two children are playing near a fountain .",
"A woman in a red coat is reading a book .",
"Several people are waiting at a bus stop .",
"A cyclist is riding through a crowded street .",
"A dog is chasing a ball on the beach .",
"A group of tourists is taking photographs .",
"A young girl is feeding ducks by a pond .",
"Two men are playing chess in a cafe .",
"A street musician is playing the violin .",
"Children are running through a field of flowers .",
"An old man is sitting on a bench reading a newspaper .",
"A woman is carrying groceries up a flight of stairs .",
"A boy is kicking a football against a wall .",
"A couple is dancing on an empty street .",
"A chef is preparing food in an outdoor kitchen .",
"A cat is sleeping on a warm windowsill .",
"Workers are repairing a road in the rain .",
"A small boat is sailing on a calm lake .",
"Firefighters are climbing a tall ladder .",
]
# pre-computed strong WMT-class system EN->DE outputs (excellent quality)
WMT_DE = [
"Ein Mann fuehrt seinen Hund im Park spazieren .",
"Zwei Kinder spielen in der Naehe eines Brunnens .",
"Eine Frau in einem roten Mantel liest ein Buch .",
"Mehrere Personen warten an einer Bushaltestelle .",
"Ein Radfahrer faehrt durch eine belebte Strasse .",
"Ein Hund jagt einen Ball am Strand .",
"Eine Gruppe von Touristen macht Fotos .",
"Ein kleines Maedchen fuettert Enten an einem Teich .",
"Zwei Maenner spielen Schach in einem Cafe .",
"Ein Strassenmusiker spielt Geige .",
"Kinder laufen durch ein Blumenfeld .",
"Ein alter Mann sitzt auf einer Bank und liest eine Zeitung .",
"Eine Frau traegt Einkaeufe eine Treppe hinauf .",
"Ein Junge kickt einen Fussball gegen eine Wand .",
"Ein Paar tanzt auf einer leeren Strasse .",
"Ein Koch bereitet Essen in einer Aussenkueche zu .",
"Eine Katze schlaeft auf einem warmen Fensterbrett .",
"Arbeiter reparieren im Regen eine Strasse .",
"Ein kleines Boot segelt auf einem ruhigen See .",
"Feuerwehrleute steigen eine hohe Leiter hinauf .",
]
print("=== Time machine: strong scaled WMT system, EN -> DE ===")
for en, de in zip(TIME_MACHINE_EN, WMT_DE):
print("EN:", en)
print("DE:", de)
print()