AI That Thinks Fast and Slow

At a glance:

• It is important to improve LLMs' attention strategies to become more context aware and distinguish spurious vs salient patterns.
• A new paper from Meta researchers takes inspiration from theories of fast vs slow conceptualizations of human cognition. Irrelevant context increases incorrect answers in LLMs by upweighting token probabilities.
• Researchers propose a "System 2 Attention" (S2A) technique for models to deliberately regenerate relevant context to focus on.
• S2A mimics goal-driven, effortful top-down human attention—removes distraction, allows control. S2A has models regenerate stripped down context, then answer questions using that rather than original context.
• Experiments show S2A improves factuality, objectivity, and math accuracy by screening out unnecessary information.
• LLMs improve when introducing a deliberative, selective attention mechanism to overcome issues with standard soft attention through regenerating focused context.

Reasoning and planning is somewhat a holy grail in AI, and required for higher-level intelligence, sensemaking, and decision making under uncertainty. An important interim step on the route to a general intelligence is to refine a machine’s attention strategies so that AIs are more context aware, hallucinate less, and are better able to distinguish spurious correlations from salient patterns in text. Think of this as being able to direct your attention in such a way as to distinguish what matters about the world from what merely appears in front of you. You see trees but what matters is that there is a forest fire coming over the ridge.