Interpreting Intelligence Part 3

Key Points:

• Grokking Phenomenon: In 2021, researchers discovered “grokking,” where models suddenly switch from memorizing training data to generalizing on unseen inputs after extended training. This phenomenon has since been replicated at larger scales.
• Three-Stage Generalization Process: Generalization involves three stages: initial memorization, formation of intricate internal circuits for problem-solving, and a “clean up” phase where redundant data dependencies are shed. This gradual process underlies the seemingly sudden shift in performance metrics.
• Phase Transition in Learning: The abrupt shift from memorization to generalization is akin to a phase transition, where the model experiences a sudden leap in performance. This occurs because memorization becomes increasingly complex, prompting the model to adopt a simpler, generalizable solution.
• Occam’s Razor for AI: The transition is driven by the model’s inherent bias towards simpler solutions, preventing further memorization and favoring generalization to handle complexity more efficiently.
• Human-Like Learning: This transition mirrors human learning, where memorization lays the foundation for understanding general patterns. Generalization allows intelligence to respond to new situations, making it fundamental to human creativity and adaptability.
• Redefining Intelligence: The ability to generalize, to learn and apply knowledge to new situations, is what defines intelligence. This quality makes human intelligence multifaceted and useful, and observing it in AI models is particularly striking.

This week, in part 3, we look at what we've learned this year since the discovery in 2021 of "grokking" where generalization happens abruptly and long after fitting the training data.

You can read part 1 of the series here, and part 2 here.

Switching from Memorizing to Generalizing

In 2021, researchers training tiny models made a surprising discovery. A set of models suddenly flipped from memorizing their training data to correctly generalizing on unseen inputs after being trained for a much longer time. Since then, this phenomenon—called “grokking”—has been investigated further and reproduced in many contexts, at larger scale.

Generalization is a three stage process. Initially, models memorize data. They then form intricate internal circuits for problem solving. Finally, they refine these solutions. In a “clean up” phase, they shed redundant data dependencies.

Though appearing sudden in performance metrics, this process is gradual and nuanced under the surface. Train versus test metrics, which track the learning over time, show a linear progression. The sudden shift is evidence of the complex, layered nature of AI learning, where transformative moments are built upon a foundation of gradual, consistent learning.