What Is ChatGPT Doing... and Why Does It Work?

Core Framework

Wolfram breaks ChatGPT's operation into three stages:

1. Tokenization and embedding: The prompt splits into tokens, then converts to numerical vectors
2. Neural network processing: Vectors pass through transformer layers that modify and create new embeddings
3. Next-token prediction: The network evaluates thousands of possibilities to generate output that captures human-like nuance

Key Concepts

• Embeddings: Dense numerical representations that capture semantic meaning—words with similar meanings cluster together in vector space
• Attention mechanism: How transformers weigh relationships between all tokens in context, enabling long-range dependencies
• Training vs. architecture: Almost nothing is explicitly engineered; the model learns everything from training data given only the overall structure
• Semantic laws of motion: Wolfram speculates that language follows discoverable rules governing how meaning flows and transforms

The Paradox of Understanding

ChatGPT reveals a tension: we built something that works remarkably well, yet we lack a clear theory explaining why it works. The model captures patterns in human language without anyone programming those patterns explicitly. This raises questions about whether computation can truly model thought, or whether we've stumbled onto something deeper about the structure of language itself.