Large Language Models flashcards, generated for you.

LLMs learn statistical patterns from training data, then predict the next word by calculating probabilities: 'given all previous words, which word is most likely to come next?' They repeat this process to generate fluent text.

The probability of the next word depends heavily on distant words, not just the immediate previous one. LLMs must track relationships across long sequences to predict sensibly—e.g., 'The bank executive sat at her...' requires remembering 'bank' (not river) to predict 'desk' instead of 'shore'.

Earlier RNNs processed words sequentially, making it hard to connect distant words and slow to train. Transformers use attention: they directly compare each word to ALL other words in parallel, letting the model efficiently learn long-range relationships.

Attention lets the model ask 'which previous words are relevant to understanding the current word?' and automatically learn to focus on the right ones.

Softmax converts raw attention scores (which can be any size) into a probability distribution: all scores sum to 1, higher scores become larger probabilities, and the model learns to concentrate attention weight on relevant words.

Attention(Q, K, V) = softmax(QK^T / √d_k) V, where Q (queries) and K (keys) are compared to find relevance, V (values) are the actual information retrieved, and √d_k prevents the dot products from becoming too large.

Dot products of high-dimensional vectors grow very large, pushing softmax into flat regions where gradients vanish during training. Scaling down by √d_k keeps attention scores in a reasonable range, preserving useful gradient flow.

Multi-head attention runs several independent attention mechanisms in parallel, allowing the model to simultaneously attend to different types of relationships (e.g., one head tracks grammar, another tracks semantic meaning, etc.) and combine insights.

Positional encodings (typically sinusoidal functions of position) are added to word embeddings before attention. They encode 'which position in the sequence,' allowing the model to learn that word order matters while still using parallel computation.

During training, the model sees all tokens at once (teacher forcing). During inference, it generates one token at a time, feeding its own predictions back as input. This mismatch can cause errors to accumulate—a problem called exposure bias that fine-tuning helps mitigate.