How Large Language Models Work

Large Language Models (LLMs) are the foundation of modern Generative AI. Understanding how they work helps you use them more effectively and build better applications.

What is an LLM?

An LLM is a neural network trained on massive text datasets to predict the next token in a sequence. Despite this simple objective, the resulting models can write essays, solve math problems, generate code, and reason about complex topics.

The Transformer Architecture

LLMs are built on the Transformer architecture, introduced in 2017. Here's a simplified view:

Input Text → Tokenizer → Embeddings → Transformer Blocks → Output Probabilities
                                            ↑
                                    (Self-Attention +
                                     Feed-Forward Layers)
                                     × N layers

Key Components

1. Tokenizer: Splits text into tokens (subword units)
2. Embedding Layer: Converts tokens into numerical vectors
3. Transformer Blocks: Stack of attention + feed-forward layers
4. Output Layer: Produces probability distribution over vocabulary

Tokenization

LLMs don't process raw text — they work with tokens, which are subword units.

Input:  "Hello, how are you doing today?"

Tokens: ["Hello", ",", " how", " are", " you", " doing", " today", "?"]

Common tokenization methods:

• BPE (Byte Pair Encoding): Used by GPT models
• SentencePiece: Used by Llama, Gemini
• WordPiece: Used by BERT

Important implications:

• 1 token ≈ 4 characters or ¾ of a word in English
• Rare words use more tokens than common words
• Code and non-English text may tokenize less efficiently
• Token limits determine how much text a model can process

Self-Attention Mechanism

Self-attention is the core innovation that makes transformers powerful. It allows each token to "attend to" every other token in the sequence.

How Attention Works

For each token, the model computes:

1. Query (Q): "What am I looking for?"
2. Key (K): "What do I contain?"
3. Value (V): "What information do I provide?"

Sentence: "The cat sat on the mat because it was tired"

When processing "it":
  → High attention to "cat" (it refers to the cat)
  → Low attention to "mat" (less relevant)
  → Medium attention to "tired" (describes the state)

This allows the model to understand context and relationships across the entire input.

Multi-Head Attention

Instead of one attention calculation, transformers use multiple "heads" in parallel. Each head can focus on different types of relationships:

• Head 1: Subject-verb agreement
• Head 2: Pronoun resolution
• Head 3: Semantic similarity
• Head 4: Positional proximity

Text Generation: Next Token Prediction

LLMs generate text one token at a time by predicting the most probable next token:

Prompt:    "The capital of France is"
Step 1:    "The capital of France is" → [Paris: 95%, Lyon: 2%, ...]
Step 2:    "The capital of France is Paris" → [.: 80%, ,: 15%, ...]
Step 3:    "The capital of France is Paris." → [It: 30%, The: 25%, ...]

Temperature and Sampling

Temperature controls randomness in generation:

Other sampling strategies:

• Top-k: Only consider the k most likely tokens
• Top-p (nucleus): Consider tokens until cumulative probability reaches p
• Repetition penalty: Reduce probability of recently used tokens

Training Process

Phase 1: Pre-training

The model reads massive text corpora and learns to predict the next token:

• Data: Books, websites, code, academic papers (trillions of tokens)
• Objective: Minimize prediction error across all training text
• Result: General language understanding and world knowledge
• Cost: Millions of dollars in compute

Phase 2: Fine-tuning (Instruction Tuning)

The base model is refined on curated instruction-response pairs:

Instruction: "Summarize this article in 3 bullet points"
Response: "• Point 1... • Point 2... • Point 3..."

Phase 3: RLHF (Reinforcement Learning from Human Feedback)

Human raters rank model outputs, and the model learns to prefer responses humans rate highly. This is what makes models helpful, harmless, and honest.

Prompt: "How do I pick a lock?"

Before RLHF: [Detailed lockpicking instructions]
After RLHF:  "I can help with legitimate locksmith needs.
              If you're locked out, contact a professional..."

Context Window

The context window is the maximum number of tokens a model can process at once (input + output combined).

Larger context windows allow:

• Processing longer documents
• More conversation history
• Fewer hallucinations from missing context

Emergent Abilities

As models scale, they develop capabilities not explicitly trained for:

• In-context learning: Learning from examples in the prompt
• Chain-of-thought reasoning: Step-by-step problem solving
• Code generation: Writing functional programs
• Translation: Converting between languages
• Instruction following: Understanding complex multi-step instructions

Limitations

Understanding limitations is as important as understanding capabilities:

1. Hallucinations: Models can confidently generate false information
2. Knowledge cutoff: Training data has a date boundary
3. No real understanding: Pattern matching, not true comprehension
4. Context window limits: Can't process infinitely long inputs
5. Bias: Models reflect biases present in training data
6. Cost: Large models are expensive to run

Summary

LLMs work by:

1. Converting text to tokens
2. Processing tokens through transformer layers with self-attention
3. Predicting the next most likely token
4. Repeating until the response is complete

This simple mechanism, scaled to billions of parameters and trillions of training tokens, produces remarkably capable AI systems. In the next topics, we'll learn how to effectively harness these capabilities.