Transformers flashcards, generated for you.

Transformers can process entire sequences in parallel and learn long-range dependencies without being constrained by sequential order, enabling efficient training on massive text datasets.

Attention allows the model to dynamically focus on different parts of the input sequence when processing each word, rather than treating all words equally or relying on sequential memory.

Queries ask 'what am I looking for?', keys describe 'what is available to look at', and values are 'what information do I get'. The model matches queries to keys to decide which values to retrieve.

Softmax converts raw attention scores into a probability distribution (normalized, sum to 1) that determines how much weight each key-value pair receives, enabling the model to focus selectively.

Multiple attention heads let the model attend to different representation subspaces simultaneously. Each head can learn different types of relationships (e.g., syntax vs. semantics), increasing expressiveness.

Attention(Q, K, V) = softmax(QK^T / √d_k)V, where Q, K, V are query, key, value matrices; d_k is key dimension; scaling by √d_k prevents vanishingly small gradients.

Positional encodings inject word position information into embeddings using sine/cosine functions at different frequencies. They're necessary because attention operates on set-like operations that are permutation-invariant and don't inherently encode sequence order.

A feed-forward network (typically two linear layers with ReLU activation) processes each position independently after attention. It adds non-linearity and allows the model to reason about attended information through learned transformations.

Layer norm is applied after attention and feed-forward layers (or before, in pre-LN architecture). It stabilizes training by normalizing activations across features, reducing internal covariate shift and enabling deeper, more stable networks.

A causal mask prevents attention from looking at future tokens by setting future positions to -∞ before softmax. It's critical for autoregressive language models so prediction of token i only uses tokens 0 to i-1, preserving causality.