Natural Language Processing flashcards, generated for you.

To enable computers to understand, interpret, and generate human language in a meaningful and useful way, bridging the gap between raw text and actionable meaning.

Words have multiple meanings (polysemy), phrases can be structured multiple ways (syntax), and meaning depends on context and culture. Humans resolve this effortlessly; machines must learn patterns from data.

Breaking text into individual units (tokens: words, subwords, or characters). It converts unstructured text into discrete pieces that algorithms can process mathematically.

Word embeddings map words to dense vectors where similar words are close together in space. One-hot encoding treats all words as equally different; embeddings capture semantic relationships (e.g., 'king' - 'man' + 'woman' ≈ 'queen').

Transformers use self-attention to directly compare every word to every other word in parallel, rather than processing sequentially. This enables faster training and better capture of long-range dependencies.

Attention(Q,K,V) = softmax(QK^T/√d_k)V. Q matches against all K vectors to produce alignment weights (softmax), then weights are used to aggregate V vectors. This determines which words to focus on.

Softmax converts raw alignment scores into a probability distribution (0 to 1, summing to 1). It emphasizes high-scoring matches while suppressing low ones, enabling interpretable attention weights.

Transformers process all tokens in parallel and don't have inherent sequential order. Position encodings inject word order information into embeddings so the model knows which word came first. RNNs naturally preserve order through sequential processing.

Pre-training: learn general language patterns on massive unlabeled data (next token prediction, masked language modeling). Fine-tuning: adapt the pre-trained model to a specific task with labeled data. This is dominant because pre-training captures universal linguistic knowledge, dramatically reducing data and compute needed for downstream tasks.

Language models predict the next token by sampling from P(token|previous context). They hallucinate when patterns learned during training lead to confident predictions that are factually false—the model optimizes for linguistic plausibility, not truth.