Deep Learning flashcards, generated for you.

Deep learning uses layers of artificial neurons (stacked transformations) to automatically discover patterns in raw data. Each layer learns hierarchical features—low layers detect simple patterns (edges), high layers combine them into complex concepts (objects).

A neuron takes multiple inputs, weights them by importance, sums them, and applies a non-linear function to produce an output. It mimics biological neurons: inputs are like dendrites, weights are synaptic strength, and the activation function introduces the ability to learn non-linear patterns.

Without activation functions, stacking layers would just multiply matrices—mathematically equivalent to one layer. Activation functions introduce non-linearity, enabling networks to learn curved decision boundaries and complex patterns that linear transformations alone cannot express.

Backpropagation computes how much each weight contributed to the final error by applying the chain rule backward through layers. It enables efficient learning by calculating gradients ∂Loss/∂weight, telling us which direction and magnitude to adjust each weight to reduce error.

A loss function quantifies the difference between predicted and actual outputs (e.g., mean squared error for regression, cross-entropy for classification). It converts abstract performance into a single number we can minimize, guiding the optimization process.

Imagine standing on a foggy hill trying to reach the lowest valley. Gradient descent calculates the slope beneath you (gradient) and takes small steps downhill. Each step moves in the direction of steepest descent: weight ← weight − learning_rate × gradient.

In very deep networks, gradients are multiplied backward through many layers via the chain rule. If each layer's gradient is <1, these multiply together exponentially, making gradients near zero in early layers. Early layers learn almost nothing—the 'vanishing gradient problem.' (Solved partly by ReLU and skip connections.)

Underfitting: model is too simple, fails to learn the training data well (high bias, high training loss). Overfitting: model memorizes training data including noise, generalizes poorly to new data (high variance, low training loss, high test loss). Regularization and early stopping combat overfitting.

Convolutional layers use small, sliding filters (kernels) that share weights across the image. This respects spatial locality: filters detect features like edges anywhere in the image. Dense layers treat each input as independent, missing spatial structure. Convolutions are more efficient and translation-invariant.

Transformers use self-attention: each token learns which other tokens to focus on, weighted by learned relevance scores. Unlike RNNs (sequential, slow), transformers process all tokens in parallel. Unlike CNNs (fixed receptive field), attention dynamically connects any tokens regardless of distance, enabling better long-range dependencies.