72 · Benchmarks — Linguini, DecipherBench

72 · Benchmarks — Linguini, DecipherBench#

Dependencies: sacrebleu
Runtime: < 5 min CPU

Chapter stub — implementation coming in a future commit.

Time-machine sentences#

Every chapter runs the same 20 held-out test sentences so the quality arc across chapters is directly comparable.

Open In Colab

# Colab setup: install this part's pinned dependencies (skipped outside Colab).
import sys
if "google.colab" in sys.modules:
    get_ipython().system("wget -q -O requirements_part7.txt https://raw.githubusercontent.com/eduardosanchezg/mthistory/main/requirements_part7.txt")
    get_ipython().system("pip install -q -r requirements_part7.txt")

72 · Benchmarks — Low-Resource and Massively Multilingual#

The benchmark landscape has evolved from bilingual WMT to massively multilingual evaluations targeting truly low-resource languages.

1 * The benchmark landscape#

benchmarks = [
    ("WMT",       "Annual", "~10-30", "News",         "High-resource pairs"),
    ("FLORES-200","Static", "200",    "Wikipedia/News","All 39K directions"),
    ("OPUS-100",  "Static", "100",    "Mixed",         "Balanced per-pair"),
    ("Linguini",  "Static", "~50",    "Puzzles",       "Reasoning-based"),
    ("Biblija",   "Static", "~100",   "Religious",     "Low-resource focus"),
]

print("{:<14} {:<8} {:<8} {:<16} {}".format(
    "Benchmark","Updated","Langs","Domain","Notes"))
print("-" * 75)
for row in benchmarks:
    print("{:<14} {:<8} {:<8} {:<16} {}".format(*row))

2 * Linguini benchmark#

print("Linguini: language-independent reasoning puzzles")
print()
print("Puzzle format: given a few example translations of a low-resource")
print("language, infer its grammar rules and translate a new sentence.")
print()
print("Example (Lao-style toy language):")
print()
puzzle = {
    "examples": [
        ("mak kin khao",   "I eat rice"),
        ("mak kin pla",    "I eat fish"),
        ("kao kin khao",   "He eats rice"),
        ("mak deum nam",   "I drink water"),
    ],
    "test": "kao deum nam",
    "answer": "He drinks water",
}

print("Given examples:")
for src, tgt in puzzle["examples"]:
    print("  {:<20} -> {}".format(src, tgt))
print()
print("Translate:", puzzle["test"])
print("Answer:   ", puzzle["answer"])
print()

vocab = {}
for src, tgt in puzzle["examples"]:
    for s, t in zip(src.split(), tgt.split()):
        vocab[s] = t

predicted = " ".join(vocab.get(w, "?") for w in puzzle["test"].split())
print("Simple word-mapping prediction:", predicted)
print("(Correct! Word order matches, morphology abstracted away.)")

3 * DecipherBench – toy decipherment#

print("DecipherBench: translate from an unseen language using only a few")
print("parallel examples -- tests reasoning, not memorisation.")
print()

# Toy cipher-language: each English word shifted by 3 in a vocab
WORD_LIST = ["a","the","man","woman","dog","cat","runs","walks",
             "quickly","slowly","in","park","garden","small","large"]
CIPHER = {w: WORD_LIST[(i+3) % len(WORD_LIST)] for i, w in enumerate(WORD_LIST)}
DECIPHER = {v: k for k, v in CIPHER.items()}

cipher_examples = [
    ("dog runs quickly", " ".join(CIPHER.get(w,"?") for w in "dog runs quickly".split())),
    ("a man walks",      " ".join(CIPHER.get(w,"?") for w in "a man walks".split())),
    ("the woman runs",   " ".join(CIPHER.get(w,"?") for w in "the woman runs".split())),
]

test_en = "a dog walks slowly"
test_cipher = " ".join(CIPHER.get(w,"?") for w in test_en.split())

print("Cipher-language examples (few-shot):")
for en, cipher in cipher_examples:
    print("  English: {:<25} Cipher: {}".format(en, cipher))
print()
print("Test:", test_cipher)

# Infer mapping from examples
mapping = {}
for en, cipher in cipher_examples:
    for ew, cw in zip(en.split(), cipher.split()):
        mapping[cw] = ew

predicted = " ".join(mapping.get(w, "[" + w + "?]") for w in test_cipher.split())
print("Predicted:", predicted)
print("Correct:  ", test_en)
print("Match:", predicted == test_en)

4 * Coverage vs depth tradeoff#

tradeoff = [
    ("WMT",        30,   "High",     "Deep",    "Detailed human eval, few langs"),
    ("FLORES-200", 200,  "Low-High", "Shallow", "All directions, 1012 sents"),
    ("OPUS-100",   100,  "Low-High", "Shallow", "Balanced pairs"),
    ("Linguini",   50,   "Low",      "Deep",    "Reasoning, not MT quality"),
]

print("{:<14} {:<7} {:<12} {:<10} {}".format(
    "Benchmark","Langs","Resource","Depth","Notes"))
print("-" * 72)
for row in tradeoff:
    print("{:<14} {:<7} {:<12} {:<10} {}".format(*row))
print()
print("No single benchmark covers all dimensions.")
print("Best practice: use FLORES-200 for coverage, WMT for depth.")

5 * Measuring low-resource progress#

import math

# Illustrative BLEU scores for 10 low-resource languages over 3 years
langs = ["sw","ha","yo","ig","am","ti","ln","kg","ts","ny"]
bleu_2020 = [4.2, 3.1, 5.0, 2.8, 6.1, 1.9, 2.3, 1.5, 3.8, 4.5]
bleu_2024 = [18.3,14.7,21.2,12.5,23.1, 9.4,10.8, 7.2,16.4,19.8]

print("Low-resource progress 2020->2024 (FLORES-200 EN->XX BLEU):")
print()
print("{:<6} {:>7} {:>7} {:>8}".format("Lang","2020","2024","Gain"))
print("-" * 34)
for lang, b20, b24 in zip(langs, bleu_2020, bleu_2024):
    gain = b24 - b20
    print("{:<6} {:>7.1f} {:>7.1f} {:>+8.1f}".format(lang, b20, b24, gain))

mean20 = sum(bleu_2020)/len(bleu_2020)
mean24 = sum(bleu_2024)/len(bleu_2024)
var20 = sum((x-mean20)**2 for x in bleu_2020)/len(bleu_2020)
var24 = sum((x-mean24)**2 for x in bleu_2024)/len(bleu_2024)
print("-" * 34)
print("{:<6} {:>7.1f} {:>7.1f} {:>+8.1f}".format("mean",mean20,mean24,mean24-mean20))
print("{:<6} {:>7.1f} {:>7.1f}".format("std",math.sqrt(var20),math.sqrt(var24)))
print()
print("Aggregate gain: +{:.1f} BLEU avg. Variance stayed high.".format(mean24-mean20))
print("Aggregate metrics hide per-language disparity -- always report per-language.")

6 * The frontier#

What remains hard:

  • Truly low-resource: <1K pairs, many African/Indigenous languages

  • Reference-free evaluation: how to evaluate without a reference translation?

  • Dialectal variation: Arabic dialects, regional Spanish, etc.

  • Domain shift: models trained on news fail on medical or legal text

  • Long documents: coherence, coreference, discourse structure

7 * Course wrap-up#

We have traced machine translation from Claude Shannon’s 1948 entropy calculations through Warren Weaver’s 1949 cryptanalysis analogy, the rule-based Georgetown demo (1954), Nagao’s example-based approach (1984), the IBM statistical models (1990), phrase-based SMT (2003), the neural revolution (2013-2014), the Transformer (2017), massively multilingual models (2019-2022), and LLM-based MT (2023+).

Three things have remained constant across 75 years:

  1. The core problem: map sequences of symbols across languages.

  2. The evaluation challenge: defining and measuring quality is hard.

  3. The gap: low-resource and under-documented languages remain underserved.

The time-machine cells in each chapter make the quality arc visceral: the same 20 sentences, evaluated across every era. That arc is the course.

Contents