Natural Language Processing flashcards, generated for you.

To enable computers to understand, interpret, and generate human language in a meaningful and useful way, bridging the gap between raw text and actionable insights.

Because computers work with discrete units, not continuous text. Tokenization breaks text into words, subwords, or characters so we can assign numerical values and process language systematically.

Words with similar meanings should live close together in a mathematical space. Instead of treating words as isolated symbols, embeddings represent them as vectors where proximity reflects semantic relationship.

BoW ignores word order and context—'dog bites man' and 'man bites dog' look identical. This loses crucial meaning, so modern NLP uses sequential models to capture how word order shapes interpretation.

RNNs maintain a hidden state that updates as they process each word, allowing information from earlier words to influence later predictions. Think of a reader maintaining memory of what they've read so far.

RNNs struggle to propagate learning signals through long sequences—gradients shrink exponentially backward. LSTMs add memory cells with gates (input, forget, output) that control information flow, preserving important signals over long distances.

Instead of processing sequences one step at a time, Transformers process all words simultaneously and use attention to weight which words are most relevant to each other. This parallelization is faster and captures long-range dependencies more effectively.

For each word (query), attention computes a weighted sum of all words (values) proportional to their similarity (dot product with query and key). Formula: Attention(Q,K,V) = softmax(QK^T/√d_k)V, producing a context vector.

Pre-training forces models to learn general linguistic patterns, grammar, world knowledge, and reasoning from unlabeled data. Fine-tuning then adapts this foundation to specific tasks with minimal labeled data, achieving strong performance with efficiency.

Encoders like BERT excel at understanding/classification tasks (bidirectional context). Decoders like GPT excel at generation (predicting next token autoregressively). Encoder-decoders (T5, Transformers) combine both for sequence-to-sequence tasks like translation.