AI

Sampling in GPT-2 is almost minimalistic. Here we look directly at top-k, temperature, and the single-token autoregressive loop that drove early generation behavior.

GPT-2’s tokenizer is surprisingly readable. This article walks through how the reversible bytes to unicode mapping and BPE merge list actually work in practice.