Why alignment is a privacy problem

The pitch for a helpful assistant is that it knows your context. The cost of it knowing your context is that your context now exists somewhere as a record.

Personalization is retention

Every technique that makes a model better at serving a specific person — memory, profiles, preference learning, long-horizon logs — is a technique for accumulating a dossier. The alignment work and the surveillance work are, structurally, the same work with different framing.

The way out is not to stop personalizing. It is to keep the personalization on the user’s side of the boundary, where deleting it means something.