01 · Weaver (1949) — The Cryptanalysis Window

01 · Weaver (1949) — The Cryptanalysis Window#

Dependencies: stdlib only — collections, math, random Runtime: < 5 seconds on any CPU

In July 1949 Warren Weaver circulated a memorandum titled simply Translation. It is the founding document of machine translation: it took Shannon’s brand-new information theory (Chapter 00) and asked whether translation could be done mechanically. Weaver floated four ideas — meaning from context, the logical basis of language, cryptographic translation, and language universals. This chapter implements the most concrete and influential one: the context window.

Weaver’s “opaque mask”#

“If one examines the words in a book, one at a time as through an opaque mask with a hole in it one word wide, then it is obviously impossible to determine, one at a time, the meaning of the words. […] But if one lengthens the slit in the opaque mask, until one can see not only the central word in question but also say N words on either side, then if N is large enough one can unambiguously decide the meaning of the central word. […] The practical question is: ‘What minimum value of N will, at least in a tolerable fraction of cases, lead to the correct choice of meaning for the central word?’”

— Weaver, Translation (1949)

That last sentence is an experiment, and we run it. We build a statistical word-sense disambiguator that sees only a window of ±N words, then sweep N to find the smallest window that reliably picks the correct sense — and therefore the correct translation.

This is, in miniature, the algorithm that Gale, Church & Yarowsky would formalise as statistical WSD four decades later. Weaver saw it in 1949.

# This chapter is **stdlib only** — nothing to install.
# It runs on any Python 3.8+ with no pip dependencies.

1 · The ambiguity#

A word like bank is polysemous: its English form hides two meanings that translate to two completely different German words.

English	Sense	German	Disambiguating context
bank	FINANCE	Bank	money, account, loan, cash, …
bank	RIVER	Ufer	river, water, fish, boat, …
plant	FACTORY	Werk	factory, workers, machines, …
plant	FLORA	Pflanze	garden, soil, leaves, grow, …
spring	SEASON	Frühling	summer, flowers, warm, …
spring	COIL	Feder	metal, steel, tension, …

Through Weaver’s one-word slit, bank is unresolvable. The meaning lives in the surrounding words.

import math
import random
from collections import Counter, defaultdict

random.seed(1949)  # the year of the memorandum

# Each sense carries its German translation and the words that signal it.
SENSES = {
    "bank": {
        "FINANCE": {"de": "Bank", "context":
            ["money", "account", "loan", "cash", "deposit",
             "savings", "manager", "interest", "credit", "teller"]},
        "RIVER": {"de": "Ufer", "context":
            ["river", "water", "fish", "boat", "mud",
             "reeds", "current", "shore", "flood", "willow"]},
    },
    "plant": {
        "FACTORY": {"de": "Werk", "context":
            ["factory", "workers", "machines", "production", "steel",
             "assembly", "shift", "output", "industrial", "supervisor"]},
        "FLORA": {"de": "Pflanze", "context":
            ["garden", "soil", "leaves", "grow", "flower",
             "roots", "sun", "green", "pot", "watered"]},
    },
    "spring": {
        "SEASON": {"de": "Frühling", "context":
            ["summer", "flowers", "warm", "season", "blossom",
             "rain", "birds", "fresh", "march", "april"]},
        "COIL": {"de": "Feder", "context":
            ["metal", "steel", "coil", "tension", "compress",
             "mattress", "mechanical", "wire", "bounce", "force"]},
    },
}

# Neutral filler appears in every sentence regardless of sense — it is pure noise
# that the disambiguator must learn to ignore.
FILLER = ["the", "a", "of", "and", "in", "to", "was", "is", "saw",
          "near", "with", "very", "then", "had", "they", "we", "stood"]

2 · A labelled corpus#

Weaver was theorising; he had no corpus. We synthesise one so the principle is demonstrable end-to-end with zero dependencies. Each sentence places a target word at a random position and fills the rest with a mix of sense-specific context words and neutral filler. The sense label is known by construction, so we can measure accuracy.

The signal is deliberately noisy: only some positions carry sense words, so a ±1 window will often miss them. That is exactly what makes Weaver’s “minimum N” question non-trivial.

def make_sentence(target, sense, length=11, signal_rate=0.45):
    """Build one labelled example: a list of tokens with `target` somewhere inside."""
    sense_words = SENSES[target][sense]["context"]
    tokens = []
    for _ in range(length):
        if random.random() < signal_rate:
            tokens.append(random.choice(sense_words))
        else:
            tokens.append(random.choice(FILLER))
    pos = random.randrange(length)
    tokens[pos] = target
    return tokens, sense


def make_corpus(target, n_per_sense=400):
    data = []
    for sense in SENSES[target]:
        for _ in range(n_per_sense):
            data.append(make_sentence(target, sense))
    random.shuffle(data)
    return data


def split(data, test_frac=0.25):
    cut = int(len(data) * (1 - test_frac))
    return data[:cut], data[cut:]


# Inspect a few examples for 'bank'
for tokens, sense in make_corpus("bank")[:4]:
    print(f"[{sense:7s}] {' '.join(tokens)}")

3 · The window disambiguator#

A Naive Bayes classifier over the context window. For a target word with senses \(s\) and a window of context words \(c_1 \dots c_k\) (the words within ±N of the target), we pick

\[\hat{s} = \arg\max_s \; P(s) \prod_{i} P(c_i \mid s)\]

with add-1 (Laplace) smoothing so unseen words don’t zero out a sense. We work in log-space to avoid underflow.

The crucial knob is N, the half-width of Weaver’s slit. window(tokens, N) returns only the tokens visible through a slit of that size.

def window(tokens, N):
    """Return the up-to-2N context tokens within +/-N of the target word."""
    i = next(idx for idx, t in enumerate(tokens) if t in SENSES)
    if N == 0:
        return []
    left = tokens[max(0, i - N):i]
    right = tokens[i + 1:i + 1 + N]
    return left + right


def train_nb(train, N):
    sense_counts = Counter()
    word_counts = defaultdict(Counter)   # sense -> word -> count
    vocab = set()
    for tokens, sense in train:
        sense_counts[sense] += 1
        for w in window(tokens, N):
            word_counts[sense][w] += 1
            vocab.add(w)
    return sense_counts, word_counts, vocab


def predict_nb(tokens, model, N):
    sense_counts, word_counts, vocab = model
    total = sum(sense_counts.values())
    V = len(vocab)
    best_sense, best_logp = None, -math.inf
    for sense, n_s in sense_counts.items():
        logp = math.log(n_s / total)                       # log prior P(s)
        denom = sum(word_counts[sense].values()) + V       # Laplace denominator
        for w in window(tokens, N):
            count = word_counts[sense][w]
            logp += math.log((count + 1) / denom)          # log P(w | s)
        if logp > best_logp:
            best_logp, best_sense = logp, sense
    return best_sense


def accuracy(target, N, n_per_sense=400):
    train, test = split(make_corpus(target, n_per_sense))
    model = train_nb(train, N)
    correct = sum(predict_nb(toks, model, N) == gold for toks, gold in test)
    return correct / len(test)

4 · Weaver’s experiment: what minimum value of N?#

We sweep the slit width N from 0 (one-word mask — Weaver’s “obviously impossible” case) upward, and watch accuracy climb. N=0 collapses to the prior (≈50% — a coin flip between two balanced senses). Widening the slit lets the sense words come into view.

Ns = [0, 1, 2, 3, 4, 6, 8, 10]
targets = list(SENSES)

print(f"{'N':>3} | " + " | ".join(f"{t:>7}" for t in targets) + " |   mean")
print("-" * 56)
for N in Ns:
    accs = [accuracy(t, N) for t in targets]
    row = " | ".join(f"{a:6.1%}" for a in accs)
    print(f"{N:>3} | {row} | {sum(accs)/len(accs):6.1%}")

You should see accuracy jump from ~50% at N=0 to the high 90s by N≈3–4, then plateau. That plateau is Weaver’s answer. A slit only a few words wide already resolves almost all ambiguity — widening it further adds little. In 1949, with no computers to run this on, Weaver reasoned his way to the same conclusion.

5 · From disambiguation to translation#

The payoff: once the sense is chosen, the translation follows. Pick the sense through a ±3 window, then emit the German word attached to that sense. This is the first time in the course we produce a (single-word) translation that depends on context — the seed of everything in Parts 3–7.

def translate_target(tokens, target, N=3, n_per_sense=400):
    train, _ = split(make_corpus(target, n_per_sense))
    model = train_nb(train, N)
    sense = predict_nb(tokens, model, N)
    return sense, SENSES[target][sense]["de"]


probes = [
    ("bank",  "we deposited the cash at the bank with the manager"),
    ("bank",  "the boat drifted near the muddy river bank and reeds"),
    ("plant", "workers at the steel plant started the assembly shift"),
    ("plant", "she watered the green plant in the garden soil"),
    ("spring","warm rain and blossom mean spring is here in april"),
    ("spring","the steel coil acts as a spring under tension"),
]

print(f"{'target':7s} {'chosen sense':13s} {'-> German':12s}  sentence")
print("-" * 80)
for target, sentence in probes:
    toks = sentence.split()
    sense, de = translate_target(toks, target, N=3)
    print(f"{target:7s} {sense:13s} -> {de:10s}  {sentence}")

6 · Time-machine cell#

Every chapter runs the shared 20 held-out test sentences so the quality arc is comparable. Weaver’s system is a disambiguator, not a full translator, so there is nothing yet to translate sentence-for-sentence — we scan the 20 sentences for any polysemous target word and report the window’s decision where one appears. (The Multi30k-style sentences contain none, which is itself the point: full-sentence translation begins with Georgetown in Chapter 10.)

TIME_MACHINE_EN = [
    "A man is walking his dog in the park .",
    "Two children are playing near a fountain .",
    "A woman in a red coat is reading a book .",
    "Several people are waiting at a bus stop .",
    "A cyclist is riding through a crowded street .",
    "A dog is chasing a ball on the beach .",
    "A group of tourists is taking photographs .",
    "A young girl is feeding ducks by a pond .",
    "Two men are playing chess in a cafe .",
    "A street musician is playing the violin .",
    "Children are running through a field of flowers .",
    "An old man is sitting on a bench reading a newspaper .",
    "A woman is carrying groceries up a flight of stairs .",
    "A boy is kicking a football against a wall .",
    "A couple is dancing on an empty street .",
    "A chef is preparing food in an outdoor kitchen .",
    "A cat is sleeping on a warm windowsill .",
    "Workers are repairing a road in the rain .",
    "A small boat is sailing on a calm lake .",
    "Firefighters are climbing a tall ladder .",
]

print("=== Chapter 01 - Time-Machine Output ===\n")
found = False
for i, sent in enumerate(TIME_MACHINE_EN, 1):
    toks = sent.lower().rstrip(" .").split()
    targets_in = [w for w in toks if w in SENSES]
    if targets_in:
        found = True
        for tgt in targets_in:
            sense, de = translate_target(toks, tgt, N=3)
            print(f"  {i:2d}. '{tgt}' -> {sense} ({de}) :: {sent}")
if not found:
    print("  No polysemous target words (bank/plant/spring) in the shared 20.")
    print("  Weaver's window resolves word sense; full-sentence translation")
    print("  begins in Chapter 10 (Georgetown-IBM, 1954).")

Summary#

Weaver’s slit = a context window of ±N words. Through a one-word mask, bank is unresolvable; widen the mask and the sense becomes determined.
A Naive Bayes classifier over that window recovers the correct sense — and hence the correct German translation (Bank vs Ufer).
Sweeping N reproduces Weaver’s central question. Accuracy saturates by N≈3–4: a small window is enough.
This is statistical WSD, proposed in 1949, decades before the data and hardware existed to run it.

Next: Chapter 02 — Bar-Hillel’s (1960) impossibility argument. He will use a sentence Weaver’s window cannot solve (“The box was in the pen.”) to argue that fully automatic high-quality translation needs world knowledge, not just context counting.

References: Weaver, W. (1949). Translation. Memorandum. Reprinted in Locke & Booth (eds.), Machine Translation of Languages (1955).