Autoregressive

AUTOREGRESSIVE GENERATION: Prompt: "The weather today is" Step 1: P(token₁ | "The weather today is") → Sample "sunny" (p=0.3) Step 2: P(token₂ | "The weather today is sunny") → Sample "and" (p=0.4) Step 3: P(token₃ | "The weather today is sunny and") → Sample "warm" (p=0.35) Step 4: P(token₄ | "The weather today is sunny and warm") → Sample "." (p=0.5) Final: "The weather today is sunny and warm." PROBABILITY CHAIN: P(sequence) = P(t₁|prompt) × P(t₂|prompt,t₁) × P(t₃|...) × ... Each token sees EVERYTHING before it! GENERATION LOOP: while not done: logits = model(context) # Full forward pass probs = softmax(logits[-1]) # Last position next_token = sample(probs) # Choose one context.append(next_token) # Add to context if next_token == EOS: break

AUTOREGRESSIVE (GPT, Claude, LLaMA): ┌───────────────────────────────────────┐ │ Input: "Translate: Hello" │ │ ↓ │ │ Step 1: Generate "Hola" (sees input) │ │ ↓ │ │ Step 2: Generate "." (sees "Hola") │ │ ↓ │ │ Output: "Hola." │ └───────────────────────────────────────┘ Pro: High quality, flexible length Con: Slow (sequential) NON-AUTOREGRESSIVE (some translation models): ┌───────────────────────────────────────┐ │ Input: "Translate: Hello" │ │ ↓ │ │ Predict all positions at once: │ │ [Hola] [.] [PAD] [PAD] │ │ ↓ │ │ Output: "Hola." │ └───────────────────────────────────────┘ Pro: Fast (parallel) Con: Lower quality, fixed length issues DIFFUSION MODELS (Midjourney, DALL-E): Not autoregressive! Generate all at once through iterative denoising. WHY AUTOREGRESSIVE DOMINATES FOR TEXT: - Language is naturally sequential - Each word depends heavily on context - Flexible output length - Can "think out loud" (chain of thought)