Flashcards · Transformers · Free
Transformers flashcards, generated for you.
Example Transformers study cards to learn from right now — then generate a full set from your own notes (plus a practice quiz) and export to Quizlet or Anki. Free, no account needed.
Example Transformers flashcards
What is the core problem that Transformers solve?
Processing sequences (like text) in parallel instead of sequentially, and allowing every word to directly attend to every other word regardless of distance.
What does 'attention' mean intuitively in Transformers?
The model learns to focus on relevant parts of the input when processing each token. It calculates a relevance score between the current token and all other tokens to decide what to pay attention to.
Why is self-attention called 'self'?
Because the attention mechanism operates within the same sequence—tokens attend to other tokens in the same input, not to external data.
What are the three components used to compute attention?
Query (Q): what am I looking for? Key (K): what do I offer? Value (V): what information do I contain? Attention = softmax(QK^T / √d_k) × V
Why divide by √d_k in the attention formula?
To prevent the dot products from becoming too large, which would push softmax into regions with tiny gradients. √d_k is the square root of the key dimension.
What is multi-head attention and why use it?
Running multiple attention operations in parallel (different learned projections of Q, K, V), then concatenating results. Allows the model to attend to different aspects of the input simultaneously.
What does the Feed-Forward Network do in a Transformer block?
After attention, applies a fully-connected network (typically: Linear → ReLU → Linear) to each token independently. Adds non-linearity and increases model capacity.
What is the purpose of positional encoding?
Since self-attention is permutation-invariant (order-agnostic), positional encodings inject information about token positions. Usually sine/cosine functions added to embeddings.
How do encoder and decoder Transformers differ?
Encoder: processes full input sequence, each token sees all other tokens (bidirectional). Decoder: processes sequentially, each token sees only earlier tokens (causal/autoregressive masking) for generation.
Why do Transformers scale better than RNNs despite more parameters?
Parallel computation: all tokens processed simultaneously vs. sequentially in RNNs. Direct attention paths solve vanishing gradient problem. Better utilizes modern hardware (GPUs/TPUs) designed for parallelism.
Make your own Transformers study set
Flashcards for related topics
Studying Transformers to build with AI? MindloomHQ turns it into real skills — structured courses, agent projects, and certificates.
Explore MindloomHQ →