Decoding Strategies
Text Generation Methods
66 SLIDES Part 3: Advanced Topics
The Decoding Dilemma: A language model outputs probabilities, not words. How you convert probabilities to text determines whether you get repetitive garbage or creative genius.
Prerequisites
- Week 5: Transformer architecture
- Week 6: Language model fundamentals (GPT)
- Understanding of probability distributions and sampling
Overview
Generate text from language models. Greedy, beam search, sampling, and nucleus decoding.
Learning Objectives
- Explain how language models convert probabilities to text
- Compare greedy decoding vs beam search vs sampling methods
- Implement temperature scaling and its effect on randomness
- Apply top-k and nucleus (top-p) filtering techniques
- Choose appropriate decoding strategies for different tasks
Key Topics
Greedy decoding
Beam search
Temperature sampling
Nucleus sampling
Key Concepts
Greedy decodingAlways pick highest probability token (fast but limited)
Beam searchTrack multiple hypotheses for better sequences
TemperatureControls randomness (low=focused, high=creative)
Top-k samplingSample from k most likely tokens only
Nucleus (top-p) samplingDynamic cutoff based on cumulative probability
Repetition penaltyPrevent degeneration and loops
Key Visualizations
Beam Search Tree
Temperature Effects
Contrastive Vs Nucleus
Decoding