Reinforcement Learning flashcards, generated for you.

An agent learns to make decisions by interacting with an environment, receiving rewards or penalties for actions, and adjusting behavior to maximize total reward over time.

A reward is the immediate signal received after one action. A return is the sum of all future rewards (often discounted), representing long-term value of a decision.

Discounting (multiplying future rewards by γ < 1) reflects that immediate rewards are more certain and valuable than distant ones, and prevents infinite returns in ongoing tasks.

A policy is a strategy or rule that maps states to actions. It defines what action the agent should take in each situation; can be deterministic (one action per state) or stochastic (probability distribution over actions).

Exploration means trying new actions to discover better rewards; exploitation means repeating known high-reward actions. RL agents must balance learning new information (explore) with using current knowledge (exploit).

The Bellman equation states: V(s) = E[R + γV(s')], meaning a state's value equals expected immediate reward plus discounted value of the next state. It decomposes long-term value recursively, enabling efficient learning algorithms.

Model-based RL learns an internal model of the environment (how actions lead to next states) and plans using it. Model-free RL learns value functions or policies directly from experience without modeling the environment.

Q(s,a) is the expected return from taking action a in state s and then following the optimal policy. Update: Q(s,a) ← Q(s,a) + α[r + γ max Q(s',a') - Q(s,a)], where α is learning rate; learns by comparing predictions to observed rewards.

Policy gradient directly learns the policy parameters by computing gradients of expected return with respect to those parameters, moving in the direction that increases reward. Unlike Q-learning, it doesn't learn value functions; it optimizes the policy directly.

Actor-critic uses two networks: the actor (policy) decides actions, and the critic (value function) evaluates those actions. It combines policy gradient (actor) stability with value function (critic) sample efficiency, reducing variance while maintaining learning stability.