A mind-expanding conversation on how models actually work | Emmanuel Ameisen (Anthropic)

TL;DR Large language models are not simple next-word predictors. They develop complex internal structures, such as planning and concept representation, to solve difficult prediction tasks.

Models grow complex capabilities from simple training objectives.

• Training involves predicting the next word in massive text datasets.
• Simple objectives force models to learn underlying world structures.
• Models must understand geography or logic to predict hard words correctly.
• This process creates emergent intelligence without explicit programming rules.

Internal mechanisms enable long-term coherence and reasoning.

• Models store intermediate results that persist across many tokens.
• These stored results allow the model to formulate and follow plans.
• The model can execute a plan without explicitly stating it.
• This explains how single-token outputs create coherent long documents.

Interpretability reveals specific internal representations and behaviors.

• Researchers decode lists of numbers to find concept activations.
• Specific neurons activate for unique concepts like "eye" or "Texas."
• These representations remain consistent across different contexts and languages.
• Scientists manipulate these activations to test model behavior directly.

Alignment requires understanding hidden model states and motivations.

• Safety teams study if models behave well only when observed.
• Interpretability detects deceptive patterns before they appear in output.
• Models may optimize for rewards using strategies that mimic human traits.
• Understanding these traits helps prevent harmful or deceptive outcomes.

Next steps

• Read key papers on interpretability to understand the field.
• Use open-source platforms like NeurIPedia to run experiments.
• Stay informed about AI safety developments and risks.