Lecture 4: Introduction to RL Algorithms, Part 1

What this lecture covers

This lecture builds the formal mathematical framework for reinforcement learning: Markov chains, Markov decision processes (MDPs), and partially observed MDPs (POMDPs), then defines the RL objective as maximizing expected cumulative reward over a trajectory distribution, in both finite- and infinite-horizon settings.

After watching, you should be able to write down the trajectory distribution induced by a policy and an MDP, state the finite- and infinite-horizon RL objectives, and explain why reinforcement learning can optimize with gradient methods even when individual rewards are discontinuous.

Key ideas

• Markov chain: defined by a state space S and transition operator T; state probabilities evolve as mu_{t+1} = T * mu_t, and the Markov property makes T a linear operator.
• Markov decision process (MDP): adds an action space A and reward function r(s,a); the transition operator becomes a tensor conditioned on both state and action.
• Partially observed MDP (POMDP): further adds an observation space O and emission probability p(o|s), so decisions are made from observations that may not fully determine the state.
• Trajectory distribution: p_theta(tau) factorizes via the chain rule and the Markov property into an initial state distribution times a product, over time, of pi_theta(a_t|s_t) * p(s_{t+1}|s_t,a_t).
• The RL objective: choose policy parameters Theta to maximize the expected sum of rewards over trajectories; this can be rewritten as a sum, over time steps, of expected rewards under the state-action marginal at each step.
• Finite vs. infinite horizon: the finite-horizon objective sums rewards over T steps; the infinite-horizon case needs either an average-reward formulation (dividing by T) or, as introduced in a later lecture, discounting, and relies on the state-action marginal converging to a stationary distribution under ergodicity and aperiodicity assumptions.
• Why gradient methods work despite discontinuous rewards: even when a reward function is discontinuous in state (e.g. +1 for staying on the road, -1 for falling off), its expectation under a smooth, differentiable policy distribution (e.g. a Bernoulli parameter Theta) is itself smooth and differentiable in Theta, which is why RL algorithms can use gradient descent on seemingly non-smooth objectives.

Walkthrough

Terminology & notation (0:16)

The lecture recaps policy notation, states, observations, and the Markov property from the imitation learning lecture, and previews that the course will cover both fully and partially observed RL.

Reward functions and definitions (4:44)

The lecture introduces the reward function as a scalar function of state and action, builds up Markov chains into MDPs and then POMDPs, and explains the transition operator as a linear (or, for MDPs, tensor) operator.

The goal of reinforcement learning (11:08)

The trajectory distribution is defined via the chain rule and Markov property, and the RL objective is stated as maximizing the expected sum of rewards over that distribution, with an aside on rewriting it as a sum of per-timestep expectations.

Finite and infinite horizon cases (17:00) and stationary distributions (18:35)

The lecture contrasts finite-horizon sums with the infinite-horizon case, introduces the average-reward formulation, and derives the stationary distribution of the augmented state-action Markov chain as the eigenvector of the transition operator with eigenvalue one, under ergodicity and aperiodicity assumptions.

Expectations and stochastic systems (23:33)

Using a driving-off-a-mountain-road example, the lecture shows that expected reward under a smooth probability distribution is differentiable even when the underlying reward function is discontinuous, explaining why gradient-based optimization works in RL.

Before you watch

• Watch the Lecture 2 imitation learning videos first for the shared notation (policies, states, observations, the Markov property).
• Comfort with basic linear algebra (matrix-vector products) and probability (chain rule, expectation) will help with the MDP and trajectory distribution derivations.

Check your understanding

1. What distinguishes a Markov decision process from a Markov chain, and a POMDP from an MDP?
2. How is the trajectory distribution p_theta(tau) factorized using the chain rule and the Markov property?
3. Why does the infinite-horizon average-reward objective require assumptions of ergodicity and aperiodicity?
4. Why can reinforcement learning use gradient descent to optimize a reward function that is itself discontinuous?