What a model remembers

Weights are not a database. People repeat this correctly and then draw the wrong conclusion from it, which is that nothing in the training set can be recovered.

What weights are is a lossy, interference-prone compression of a corpus, and lossy compression still returns some inputs verbatim. Usually the rare ones. Usually the ones that appeared more than once.

Memorization is a tail phenomenon

A string that occurs a thousand times in the corpus is not memorized so much as learned. A string that occurs three times, is high-entropy, and sits in an otherwise predictable context — a key, an address, a phone number — is exactly the kind of thing a large model will hand back if you ask it the right way.

The practical consequences are dull and important: deduplicate aggressively, hold out canaries, measure extraction rates before shipping, and treat “we did not intend to train on that” as a statement about intent rather than about what the model contains.