Andrej Karpathy Stanford Lecture: Delete Everything, Keep Graph

TL;DR Programming is shifting from writing explicit code to designing prompts for large language models. This new paradigm allows natural language to act as a general-purpose programming language.

The evolution of software paradigms

• Software 1.0 involves writing explicit algorithms and instructions for computers.
• Software 2.0 uses neural networks trained on data engines to optimize weights.
• Software 3.0 uses prompts to condition large language models to perform tasks.
• Natural language is now the hottest new programming language for this paradigm.

Large language models function as general-purpose computers

• These models predict the next word in a sequence based on vast internet data.
• You can program them by providing context and examples in a prompt.
• The model executes the task by completing the document or generating text.
• Prompts allow the model to simulate systems like virtual machines or smart home assistants.

Transformers are efficient and flexible architectures

• Attention mechanisms replace recurrent neural networks for better parallel processing.
• The architecture operates on sets of tokens rather than fixed spatial structures.
• It is highly expressive, optimizable via gradient descent, and efficient on GPUs.
• Positional encodings add sequence order information to the set-based attention mechanism.

Prompt engineering enables complex reasoning and control

• Specific phrases like "think step by step" significantly improve answer accuracy.
• Conditioning the model on high-intelligence personas yields better quality responses.
• Engineers can define system behaviors and output formats using plain English text.
• This technique allows for dynamic reconfiguration of the model at runtime.