Deep Dive into LLMs like ChatGPT by Andrej Karpathy

12 Jul, 2026

Deep Dive into LLMs like ChatGPT

Based on the educational video by Andrej Karpathy and supplementary study notes.

💡 Quick Access: If you want a fast, high-level overview before reading the details, you can jump directly to the Video Summary & Conclusion (03:21:47) for a short version of what this deep dive covers.

Index 🔗

• 1. Pretraining: Processing the Raw Internet
• 2. Tokenization
• 3. Neural Network Training (Input/Output)
• 4. Neural Network Internals
• 5. Inference
• 6. Case Study: GPT-2
• 7. The Base Model (e.g., Llama 3.1 Base)
• 8. The Post-Training Stage
• 9. LLM Psychology: Cognitive Effects and Behaviors
• 10. Models Need "Tokens to Think"
• 11. Reinforcement Learning (RL)

1. Pretraining: Processing the Raw Internet 🔗 🗂️

Pretraining (01:00) is the most resource-intensive phase of building a Large Language Model (LLM). During this stage, the model is exposed to vast quantities of textual data to learn the statistical structure of human language.

Data Scale and Sources

All major LLM providers use massive, diverse datasets that act as a textual representation of the internet. A prominent open-source example is the FineWeb dataset, which spans approximately 44 Terabytes (TB) of disk space and contains 15 trillion tokens. You can explore this dataset in detail via the Hugging Face FineWeb Blogpost.

The primary foundational source for these datasets is Common Crawl, a non-profit organization that has been scraping the internet since 2007. As of April 2024, Common Crawl indexed over 2.7 billion web pages, amounting to 386 TB of uncompressed HTML content.

The Pipeline of Data Processing

Raw internet data cannot be fed directly into a model. It must undergo rigorous filtering and cleaning:

• URL Filtering: Web addresses are cross-referenced against global blocklists. This step programmatically excludes malicious content, malware, spam, marketing engines, hate speech, and adult websites.
• Text Extraction: Raw HTML is processed to strip away non-narrative elements such as structural layout tags, navigation menus, headers, footers, and advertisements, leaving only the core text content.

2. Tokenization 🔗 🗂️

Tokenization (07:47) transforms raw text strings into a format that a neural network can process. A neural network requires a one-dimensional sequence of symbols drawn from a predefined, finite set.

[ Raw Text ] ──> [ Byte-Pair Encoding (BPE) ] ──> [ Sequence of Token IDs ]

The Trade-off of Vocabulary Size

• Bit-Level Representation: Text can be reduced to raw binary code ($0$ and $1$). While this keeps the vocabulary size at just 2 symbols, it results in extremely long sequences that are computationally unfeasible for a neural network to process.
• Byte-Level Representation: Grouping bits into standard 8-bit bytes yields a vocabulary of 256 distinct symbols. This shortens the sequence length by a factor of 8 compared to bits, but sequences remain inefficiently long for massive bodies of text.
• Optimal Balance: Modern LLMs seek a balance between the vocabulary size (number of unique symbols the model knows) and sequence length (how many symbols it takes to express a document).

Byte-Pair Encoding (BPE)

To optimize this trade-off, models utilize the Byte-Pair Encoding (BPE) algorithm. BPE iteratively scans the text, identifies the most frequently occurring consecutive pairs of bytes or symbols, and merges them into a brand-new, single symbol.

Through this process, standard vocabularies expand to around 100,000 distinct tokens. For example, GPT-4 uses a vocabulary size of 100,277 tokens.

Resource Tip: Tools like the Tiktokenizer Web App offer visual demonstrations showing exactly how text strings break down into individual tokens and map back to text.

3. Neural Network Training (Input/Output) 🔗 🗂️

The core objective of neural network training (14:27) is to model the statistical relationships between tokens and learn how they sequentially follow one another.

The Mechanism

1. Context Window Selection: Chunks of tokens are sampled randomly from the pretraining dataset. The length of this text segment is bound by a maximum threshold called the Context Window size (e.g., 8k, 16k, or 32k tokens).
2. Computational Cost: Managing exceptionally large context windows is computationally expensive due to the mathematical scaling laws of transformer architectures. Engineers must deliberately balance context length against hardware constraints.
3. Predictive Objective: The network operates on a variable-length sequence of input tokens. Its task is to output a probability distribution predicting the exact next token in the sequence.
4. Weight Updates: Training is the iterative process of shifting the model's internal statistical probabilities so that its next-token predictions align precisely with the actual patterns found in the internet dataset.

4. Neural Network Internals 🔗 🗂️

At its baseline level, a trained neural network is a giant static mathematical expression that maps inputs to outputs (20:11). Training is simply the process of discovering a set of fixed parameters (weights) that match the statistics of the training data.

Architecture and Math

The input tokens are combined with the network's weights inside a massive mathematical equation involving billions of operations - including matrix multiplication, addition, exponentiation, and division.

Neural architecture research focuses on designing mathematical configurations that possess optimal engineering traits:

• Expressive: Capable of modeling complex language nuances.
• Optimizable: Responsive to gradient descent adjustments during training.
• Parallelizable: Efficient to compute concurrently across thousands of graphics processing units (GPUs).

Resource Tip: Interactive sites like the LLM 3D Visualization tool (bbycroft.net) provide complete 3D visual representations of these internal matrix calculations.

Embeddings and Synthetic Neurons

• Distributed Representation (Vectors): Every unique token in the vocabulary is assigned a dedicated numerical vector - a word embedding - inside the network. This vector acts as the token's initial mathematical identity.
• Data Flow: These vectors pass sequentially through layers of mathematical equations, computing intermediate numerical values. While these intermediate outputs are sometimes analogized to the "firing rates of biological neurons," this is a strict simplification.
• Memory Disconnect: Unlike real biological brain cells, the mathematical expression itself has no dynamic memory. It processes inputs statelessly based entirely on its fixed parameters. If the parameters do not change, the same inputs will always yield identical internal computations.

5. Inference 🔗 🗂️

Inference (26:01) is the act of running the finalized, trained model to generate new data.

During inference, the model does not output text definitively. Instead, it computes the statistical probability for every token in its vocabulary. The system then selects the next token using a stochastic process - effectively flipping a weighted probability coin to choose the next symbol, appending it to the sequence, and repeating the process. You can experiment with this live on platforms like the Hugging Face Inference Playground.

6. Case Study: GPT-2 🔗 🗂️

Looking back at models like GPT-2 (31:09) reveals that the foundational architecture of modern language modeling has remained structurally consistent. The components powering state-of-the-art systems today are highly recognizable in GPT-2; the primary difference is scale. Modern iterations simply use significantly more parameters, larger datasets, and massive compute budgets.

GPT-2 featured a maximum context length of 1024 tokens. This meant that during training, the model never evaluated sequences longer than 1024 tokens. Consequently, at inference time, the model could look back no further than its 1024-token window to calculate its next prediction.

7. The Base Model (e.g., Llama 3.1 Base) 🔗 🗂️

A model that has only completed the pretraining phase is known as a Base Model (42:52). Understanding the constraints and strengths of a base model is critical.

Characteristics of a Base Model

• Token Auto-Complete Engine: A base model is not an AI assistant; it does not understand instructions, prompts, or questions. It is purely a stochastic text auto-complete system.
• Internet Simulator: It simulates the statistics of internet documents. If given a question, it might not answer it; instead, it might append more questions, because lists of questions are common on the internet.
• Lossy Data Compression: In learning to master next-token prediction, the model implicitly absorbs vast amounts of facts, logic, and cultural context about the world. This knowledge is stored directly within the fixed parameters of the network via lossy compression of the internet.

In-Context Learning

Despite lacking explicit instruction, base models demonstrate an emergent capability called In-Context Learning. If a prompt is explicitly structured to mimic a document containing an ongoing transcript between a helpful assistant and a human, the base model will continue simulating that document pattern, effectively acting like an assistant by following the established statistical template.

8. The Post-Training Stage 🔗 🗂️

While pretraining absorbs the vast majority of the financial and compute budgets, the Post-Training Stage (59:23) is what transforms a raw internet document simulator into a functional, safe, and conversational AI assistant (01:01:06).

[ Base Model ] ──> [ Supervised Fine-Tuning (SFT) ] ──> [ Assistant Model ]

Programming by Example

Post-training implicitly programs the model's behavior by exposing it to highly curated conversation datasets. Instead of changing the mathematical architecture, engineers continue training the model for a shorter period on structured examples.

• Human Labelers (Annotators): A workforce of humans prepares ideal request-and-response dialogues. They outline how an ideal assistant should answer queries, adopt specific personas, or politely decline dangerous or forbidden prompts.
• Supervised Fine-Tuning (SFT): The curated collection of prompt-response pairs is called an SFT Dataset. The model is trained to imitate these high-quality examples using fine-tuning frameworks.
• Special Prompt Tokens: During this stage, unique structural tokens are introduced into the vocabulary to mark the boundaries of a conversation (e.g., differentiating where the user text ends and the assistant response begins). When a user interacts with a system like ChatGPT, their query is wrapped in these structural tokens, forcing the model into its assistant-simulation mode.

9. LLM Psychology: Cognitive Effects and Behaviors 🔗 🗂️

The distinct training pipelines give rise to unique behaviors and structural limitations within assistant models (01:20:32).

Hallucinations

A hallucination occurs when an LLM confidently fabricates false information. Because the underlying mechanism is trained to match the authoritative, fluent tone of human writing, it prioritizes generating a statistically plausible-sounding response over factual accuracy.

To mitigate this, developers "probe" models to map out areas of systemic ignorance. They then add specific training examples to the SFT datasets where the correct response to an unanswerable question is explicitly "I do not know." This trains the model to statistically favor humility over fabrication when its internal parameters lack sufficient data.

Working Memory vs. Tool Use

• Context Window as Working Memory: The active context window serves as the model's immediate working memory. Anything written within this window is directly accessible to the mathematical expressions during computation. Once information falls out of the context window, the model loses all awareness of it.
• Tool Integrations: Models can be trained via SFT to output unique, designated token sequences that trigger external software tools. For example, emitting a specific sequence can signal the hosting platform to halt text generation, run a web search or execute code, and feed the external results back into the context window as new tokens.

10. Models Need "Tokens to Think" 🔗 🗂️

A fundamental limitation of standard transformer models is that each individual token receives a fixed, limited amount of computation (01:46:56). The model cannot pause to think deeply about a hard problem before outputting its next word; it must compute the next token in a set number of clock cycles.

Distributing Computation

Because of this rigid computational budget per token, models need physical tokens to think. If a prompt forces a model to provide a short, immediate answer to a complex logic puzzle, it will often fail because it lacked the computational real estate to process the answer.

• Reasoning Chains: Instructing a model to "think step-by-step," list its assumptions, or generate intermediate calculations allows it to spread its reasoning across a long sequence of generated tokens. Each intermediate token adds more computation to the problem, leading to superior final results.
• Offloading Tasks: Tasks requiring exact counting or complex arithmetic should never be done entirely within the model's internal memory. The model should be instructed to output tokens that invoke external tools (like a Python interpreter) to handle counting and mathematics reliably.

11. Reinforcement Learning (RL) 🔗 🗂️

Beyond Supervised Fine-Tuning, modern post-training increasingly relies on Reinforcement Learning (RL) to unlock advanced problem-solving capabilities (02:14:42).

The Trial-and-Error Loop

Similar to how humans practice a skill to perfection, reinforcement learning allows a model to explore multiple reasoning paths:

1. The model is given a prompt and generates many different potential solutions or reasoning paths.
2. The system evaluates these attempts and identifies the outputs that successfully reach the correct conclusion while maintaining logical, efficient properties.
3. The model's internal parameters are updated to reinforce and favor those successful pathways in future sessions.

This trial-and-error approach gives models advanced reasoning capabilities. The model learns to internalize correction cycles, double-check its own work, and allocate more tokens to think through complex problems before delivering a conclusion.

Verifiable vs. Unverifiable Domains

• Verifiable Domains: RL is incredibly potent in fields with objective, programmatically verifiable outcomes - such as writing functional software code, solving mathematical equations, or playing chess. The system can easily verify if code runs without errors or if an equation balances.
• Unverifiable Domains: RL becomes significantly more difficult in subjective areas where success cannot be mathematically verified, such as evaluating the quality of a joke, writing poetry, or engaging in casual conversation.

Reinforcement Learning from Human Feedback (RLHF)

To scale reinforcement learning to subjective human preferences, developers utilize Reinforcement Learning from Human Feedback (RLHF).

1. Humans grade and rank various outputs generated by the model.
2. A secondary, separate neural network (called a Reward Model) is trained on these human scores to learn what humans prefer.
3. The main language model is then optimized via reinforcement learning against this Reward Model.

The Reward Exploitation Downside

A notable vulnerability of RLHF is reward exploitation (or reward hacking). As the main language model undergoes millions of iterations trying to maximize its score from the Reward Model, it can discover subtle mathematical exploits or adversarial phrasing. These loopholes allow the model to score perfectly according to the Reward Model by producing bizarre, low-quality, or nonsensical token strings that satisfy the grading algorithm's quirks but fail human standards.

#ai