Transformers flashcards, generated for you.

RNNs process sequences one step at a time sequentially, creating long-range dependencies that are hard to learn. Transformers process entire sequences in parallel and can directly relate any two positions, regardless of distance.

Attention is the model's ability to focus on relevant parts of the input when processing each element. For each word, it learns to weight how important every other word is for understanding it.

Query (Q): what am I looking for? Key (K): what can I offer? Value (V): what information do I contain? For each position, Q matches against all Ks to decide which Vs to blend together.

Q·K^T measures similarity between queries and keys. Dividing by √d_k (scaling) prevents dot products from growing too large, which would crush gradients through softmax. This stabilizes training.

Attention(Q,K,V) = softmax(QK^T/√d_k)V. This computes attention weights via softmax over scaled similarities, then uses those weights to average the values.

Multiple heads allow the model to attend to different aspects simultaneously (e.g., one head tracks nouns, another tracks verbs). Each head learns different representation subspaces independently, then results concatenate.

Since Transformers process all tokens in parallel without sequential order, positional encoding injects position information by adding learnable or fixed sinusoidal patterns to embeddings. This tells the model where each word is in the sequence.

Multi-head self-attention → Add & Norm (residual + layer norm) → Feed-forward (two linear layers with ReLU) → Add & Norm. Residual connections and normalization stabilize deep stacking.

Causal masking prevents the decoder from attending to future tokens during training. Without it, the model could 'cheat' by looking ahead at answer tokens. At inference, no masking is needed; autoregressive generation handles causality naturally.

RNNs process sequentially (can't parallelize), O(n) steps with O(d²) per step. Transformers process all tokens in parallel in O(1) steps, but each step is O(n²d) due to all-pairs attention. For long sequences, Transformers are faster in wall-clock time despite higher complexity.