Overcoming Catastrophic Forgetting in Neural Networks

Proposes elastic weight consolidation, which slows learning on weights that a Fisher information estimate marks as important to previously learned tasks. It is the clearest early statement that catastrophic forgetting is a constraint problem, not a capacity problem — and the framing survives even where the method does not.

What it does

Elastic weight consolidation (EWC) adds a quadratic penalty that anchors parameters near their values after an earlier task, weighted by how much each parameter mattered:

\[ \mathcal{L}(\theta) = \mathcal{L}_B(\theta) + \sum_i \frac{\lambda}{2} F_i \left(\theta_i - \theta^{*}_{A,i}\right)^2 \]

(1)

\(F_i\) is the diagonal of the Fisher information matrix at \(\theta^{*}_A\), a cheap proxy for the curvature of the old task’s loss along each parameter direction. Directions the old task was insensitive to stay free; directions it was sensitive to become stiff.

Why it matters for general-purpose capability

A system that must keep learning after deployment cannot retrain from scratch on the union of everything it has seen. EWC reframes catastrophic forgetting as a question about which regions of parameter space are shared between tasks — which is the right question, and connects directly to OP-002.

Where it is weak

The diagonal Fisher approximation ignores parameter interactions, and the penalty accumulates: after enough tasks the model is stiff everywhere and plasticity collapses. Replay-based methods generally outperform it in practice. The contribution that lasted is the diagnosis, not the cure.