As part of the BlueDot AI Safety project programme, I studied the bidirectionality of Direct Preference Optimization (DPO), a preference-optimization method used to align models toward desired behaviour.
The motivation came from discussions in the Technical Safety course, which drew me toward alignment and interpretability. A paper by Lee et al. (arXiv:2401.01967) studies whether DPO’s edits are “shallow” when used to remove toxicity, and shows that DPO doesn’t erase the toxic capability — it learns a distributed offset that bypasses it, leaving the behaviour reactivatable.
I extended this to sycophancy — a model’s tendency to agree with a user’s stated view even when it’s wrong — and asked whether installing a behaviour and removing it travel the same path through the network. Two findings stood out. First, the sycophantic lean is already present in base models before any instruction-tuning: across both Qwen and Gemma, the pretrained model is measurably sycophantic, and instruction-tuning mainly sharpens it. Second, installing and removing sycophancy are not mirror images: in most models the two edits recruit largely distinct, distributed sets of neurons, and the magnitudes of those updates differ — undoing a behaviour is not simply “running the install in reverse.”
The question that motivated this framing comes from biology: unlearning in the brain isn’t just the deactivation of synapses that learning switched on — it often recruits new activity or different regions entirely. I wanted to know whether DPO’s unlearning shows an analogous asymmetry. My results are a first, tentative “yes,” on a small scale.
A more detailed draft is here: https://docs.google.com/document/d/1Yqub2mD6Pt17AtxyuRqvy7y-YFMIO-DygcsQhU4IcNY/edit?usp=sharing