Lecture 2: Linear Regression and Gradient Descent

What this lecture covers

This lecture works through the first full learning algorithm of the course: linear regression. It builds the notation used throughout CS229 (hypothesis, parameters, training examples, cost function), then shows two ways to find the parameters that fit a dataset: the iterative gradient descent algorithm and the closed-form normal equations. The running example is predicting house prices from house size (and later, number of bedrooms) using real listings from Portland, Oregon.

By the end, you should be able to write out the linear regression hypothesis and cost function, explain how gradient descent updates parameters step by step, distinguish batch from stochastic gradient descent, and know when the normal equations give you the answer directly instead of iterating.

Key ideas

• Hypothesis: the function h(x) the learning algorithm outputs to map inputs (house features) to a predicted output (price); for linear regression h(x) = theta_0 + theta_1*x_1 + ... + theta_n*x_n, with a dummy feature x_0 = 1 so the sum starts at j = 0.
• Parameters (theta): the values the learning algorithm chooses to make h(x) fit the training data well.
• Cost function J(theta): one-half the sum of squared differences between predicted and actual prices across all training examples; linear regression minimizes this, which is why it's also called ordinary least squares.
• Gradient descent: an iterative algorithm that repeatedly adjusts theta in the direction that decreases J(theta) fastest, controlled by a learning rate alpha.
• Batch gradient descent: each update uses the entire training set, which is accurate but slow on large datasets.
• Stochastic gradient descent: each update uses just one training example at a time, making faster but noisier progress, and is preferred for large datasets.
• Normal equations: a closed-form solution, theta = (X^T X)^-1 X^T y, that solves for the optimal theta in one step, but only works for linear regression.
• Design matrix X: the matrix formed by stacking each training example's feature vector as a row, used to write the cost function and normal equations compactly.

Walkthrough

Setting up linear regression (0:45)

Ng connects this lecture back to Monday's self-driving car demo, framing it as supervised regression, then motivates linear regression with a simpler example: predicting house prices from a dataset of Portland-area listings. He introduces the standard supervised learning pipeline: a training set is fed to a learning algorithm, which outputs a hypothesis h(x) that maps house size to a predicted price.

Notation and parameters (8:27)

Ng builds out the notation used for the rest of the course. With multiple input features (size and number of bedrooms), the hypothesis becomes h(x) = theta_0 + theta_1*x_1 + theta_2*x_2, simplified using a dummy feature x_0 = 1 so it can be written as a sum from j = 0 to n. He defines m as the number of training examples, n as the number of features, x^(i), y^(i) as the i-th training example, and theta as the parameters the algorithm must choose.

The cost function (14:44)

To choose good parameters, the lecture defines the cost function J(theta) as one-half the sum over all training examples of the squared difference between h(x) and the true label y. Minimizing squared error is called ordinary least squares; Ng notes that the choice of squared rather than absolute error will be justified later when the course covers generalized linear models.

Gradient descent (18:06)

Gradient descent starts at some initial value of theta (often all zeros) and repeatedly takes a small step in the direction that most steeply reduces J(theta), using a learning rate alpha to scale each step. Ng visualizes this as standing on a bowl-shaped surface and walking downhill, noting that the starting point can matter for functions with multiple local optima, though linear regression's cost function is quadratic and has only one (global) minimum. He then derives the update rule by taking the partial derivative of J(theta) with respect to each theta_j, arriving at theta_j := theta_j - alpha * sum((h(x^(i)) - y^(i)) * x_j^(i)). Too large a learning rate can overshoot and even increase J(theta); too small a rate makes convergence slow, so in practice you try several values, often on an exponential scale like 0.01, 0.03, 0.1.

Batch vs. stochastic gradient descent (42:34)

The version of gradient descent that sums over the entire training set on every update is called batch gradient descent. Its drawback is that with very large datasets, a single parameter update requires scanning the whole dataset, which becomes expensive. Stochastic gradient descent instead updates the parameters using just one training example at a time, looping through examples one by one. This takes a noisier, more meandering path toward the minimum and technically never fully converges, but makes much faster progress on large datasets, so it's used more often in practice. Ng mentions that decreasing the learning rate over time is a common way to shrink the oscillations as training proceeds, and briefly notes mini-batch gradient descent (using small batches of examples) as a middle ground covered in more depth in CS230.

The normal equations (53:16)

For linear regression specifically, there's a way to solve for the optimal theta directly instead of iterating. Ng introduces a compact matrix-derivative notation, then defines the design matrix X (training examples stacked as rows) and the label vector y, showing that J(theta) can be written as (1/2) * (X*theta - y)^T * (X*theta - y). Taking the derivative of this expression with respect to theta and setting it to zero yields the normal equations, X^T*X*theta = X^T*y, so theta = (X^T*X)^-1 * X^T*y. This gives the global minimum in one computation, though it doesn't generalize to the other algorithms covered later in the course. If X^T*X is not invertible, it usually means some features are linearly dependent (for example, a repeated feature).

Before you watch

• Watch Lecture 1 first for the course's framing of supervised learning and regression versus classification.
• Basic linear algebra (matrix and vector multiplication, transpose) and single-variable calculus (derivatives, partial derivatives) make the notation and the gradient descent derivation much easier to follow.

Check your understanding

1. Why does linear regression's cost function J(theta) have no local optima other than the global minimum?
2. What is the practical trade-off between batch gradient descent and stochastic gradient descent on a very large dataset?
3. What happens to gradient descent if the learning rate alpha is set too large, and how can you tell from watching J(theta)?
4. Why can't the normal equations be used for algorithms other than linear regression?