Large Language Models flashcards, generated for you.

LLMs predict the next word in a sequence by learning patterns from massive amounts of text. They build statistical models of language: given some context words, they output a probability distribution over possible next words.

Next-word prediction forces the model to learn deep semantic, logical, and factual knowledge about the world. A model that predicts well must understand meaning, causality, and reasoning—skills that transfer to many downstream tasks.

A token is a small unit of text—usually a word or subword (like 'ing' or 'ly'). Text is broken into tokens before being fed to the model; roughly 1 token ≈ 0.75 words.

The Transformer uses self-attention: each token can directly attend to (weigh and combine information from) every other token in the sequence in parallel, rather than processing sequentially. This allows efficient learning of long-range dependencies.

For each token, self-attention asks: 'Which other tokens are relevant to me?' It computes similarity scores (queries vs. keys) to decide how much to focus on each other token, then sums their values (representations) as a weighted average.

The loss measures how wrong the model's predicted probability distribution over next words is compared to the actual next word. Common loss is cross-entropy: lower loss means the model assigns higher probability to the correct next token.

Q (query) and K (key) vectors compute similarity scores via dot product: Attention = softmax(QK^T/√d) V. The softmax weights determine how much of each Value vector contributes to the output for each position. √d is a scaling factor to stabilize gradients.

A positional embedding is a learned or fixed vector added to each token's representation to encode its position in the sequence. Without it, the Transformer would treat 'dog bites man' and 'man bites dog' identically—position information is essential.

LLMs have billions of parameters (weights) and must update them to model the statistical patterns across diverse language. Scaling laws show that performance improves predictably with more data, parameters, and compute; there are no obvious shortcuts.

Larger models trained on more data exhibit emergent abilities—skills (like reasoning or coding) not explicitly trained for. These emerge suddenly at certain scales due to increased capacity to model complex patterns; smaller models cannot perform these tasks.