Lecture 7: Efficient Markets

What this lecture covers

Shiller calls the Efficient Markets Hypothesis "a half-truth" and spends the lecture explaining both why it is compelling and where it breaks down. He opens by revisiting David Swensen's guest lecture, using the question of why Swensen never cites the Sharpe ratio as a way into a discussion of how investment managers can manipulate risk-adjusted performance measures. He then traces the history of the idea that markets efficiently price all available information, from 19th-century writers through Eugene Fama's 1960s research, before testing the theory using Random Walk and mean-reverting price simulations.

This lecture builds directly on the prior guest lecture and on the course's earlier treatment of risk and diversification. After watching, you should be able to explain what the Efficient Markets Hypothesis claims, describe how a manager could fake a high Sharpe ratio, and understand the difference between a Random Walk and a mean-reverting (autoregressive) price process, along with why that distinction matters for whether markets can be beaten.

Key ideas

• Efficient Markets Hypothesis: the claim that market prices already incorporate all public information, so that consistently beating the market should be impossible.
• Sharpe ratio manipulation: a manager can appear to have a high Sharpe ratio while secretly taking on rare, catastrophic tail risk, for example by selling out-of-the-money options to collect steady premiums while exposing investors to occasional large losses; Shiller cites research by Goetzmann, Ibbotson, Spiegel and Welch on this strategy and the real-world collapse of Integral Investment Management.
• Random Walk: a price process where each period's value equals the prior value plus unforecastable noise, meaning past prices carry no information about future direction.
• First-order autoregressive (AR-1) process: a mean-reverting alternative where prices drift back toward a trend at a rate set by a coefficient rho; when rho equals 1 the process becomes a Random Walk.
• Historical roots of the theory: 19th-century writers including George Gibson (1889) and Charles Conant (1904) described markets as aggregating the judgment of the best-informed traders, well before the term "efficient markets" existed.
• The CRSP revolution: the Center for Research in Security Prices, funded by the Ford Foundation in 1960, assembled a clean historical stock database that let Eugene Fama and others test market efficiency empirically for the first time.
• Technical analysis and Head and Shoulders patterns: chart-based prediction methods promoted by analysts like McGee; Shiller argues these are not baseless but also not reliable enough to build a strategy on.
• Simulation evidence: computer-generated Random Walk series look strikingly similar to actual long-run stock price history, showing how easily people mistake randomness for meaningful pattern.

Walkthrough

Revisiting Swensen and the Sharpe ratio question (0:01)

Shiller opens by restating the Efficient Markets Hypothesis in its strongest form: the market knows more than any individual investor, so trying to beat it is futile. He then contrasts this with David Swensen's record of apparently consistent outperformance since 1985, and revisits a student's question about why Swensen never mentioned the Sharpe ratio. Swensen had argued that standard deviation is hard to measure for illiquid assets like private equity and real estate, which are appraised infrequently and therefore look artificially stable.

How managers can fake a high Sharpe ratio (5:06)

Shiller goes further than Swensen's explanation, describing a strategy documented by Goetzmann, Ibbotson, Spiegel and Welch in which a manager sells the upper tail of a return distribution (through options such as out-of-the-money calls and puts) to collect steady income while loading up the risk of a rare, severe loss. This can produce a high Sharpe ratio for years before a crash wipes out investors, as happened to the hedge fund Integral Investment Management, whose collapse in 2001 cost the Art Institute of Chicago most of a $43 million investment. Shiller connects this to disclosure law, which requires managers to actively flag material risks rather than rely on boilerplate warnings, and argues that judging an investment manager ultimately comes down to character and disclosed intentions, not just statistics.

The history of the Efficient Markets idea (16:24)

Shiller traces early statements of market efficiency to George Gibson's 1889 book, which described the stock market as a voting mechanism where informed traders' judgment sets prices, and to Charles Conant's 1904 book arguing that stock markets are essential for allocating capital, not mere gambling. He notes that the term "Efficient Markets" was coined later, by University of Chicago professor Harry Roberts, and popularized by Eugene Fama. He describes how the 1960 creation of the CRSP historical stock price database at the University of Chicago, funded by the Ford Foundation, enabled the first large-scale empirical tests of market efficiency, leading to Fama's influential 1969 review article and the high point of Efficient Markets enthusiasm in the 1970s.

Textbooks change their tune (32:42)

Comparing older and newer editions of the Brealey and Myers finance textbook he has used for 25 years, Shiller shows that a 1984 edition flatly stated security prices "reflect the true underlying value of assets," while the 2008 edition instead says more research is needed to understand why asset prices sometimes diverge from fundamentals. He draws a parallel to a New Yorker article on how initially strong scientific findings, including for certain psychiatric drugs, tend to weaken on replication, suggesting a similar dynamic of early enthusiasm followed by more sober reassessment applies to the Efficient Markets Hypothesis.

Technical analysis and the Random Walk (40:51)

Shiller reviews technical analysis, the practice of predicting prices from chart patterns, citing the classic Edwards and McGee text and its "Head and Shoulders" pattern. He notes that Burton Malkiel's influential book "A Random Walk Down Wall Street" dismissed technical analysis as unsupported by evidence, though Shiller found the cited studies didn't fully engage with Edwards and McGee's specific claims. He then defines the Random Walk formally as a series where each value equals the prior value plus random noise, tracing the term to statistician Karl Pearson's 1905 "drunkard's walk" analogy, and contrasts it with the mean-reverting AR-1 process, in which a coefficient rho between -1 and 1 pulls prices back toward a trend, with rho equal to 1 reducing the AR-1 to a pure Random Walk.

Simulating Random Walks against real stock prices (53:25)

Using a spreadsheet random number generator, Shiller repeatedly generates simulated Random Walk price paths and compares them visually to the actual S&P composite index since 1871, showing that purely random series can look just as trendy, bumpy or pattern-laden as real market history, which is why humans are easily fooled into seeing structure where none exists. He then generates AR-1 simulations with a strongly mean-reverting coefficient and shows these look visibly different from the real data, hugging a trend line far more tightly than actual prices do, which argues against strong mean reversion. He closes by noting that a very slowly mean-reverting AR-1 (rho near 1, such as 0.98 or 0.99) is nearly indistinguishable from a Random Walk and could still allow profit opportunities, but only over very long horizons of a decade or more, reinforcing his conclusion that the Efficient Markets Hypothesis is right about the difficulty of short-run profit but wrong to claim markets are perfectly efficient.

Before you watch

• Watching the prior lecture, David Swensen's guest talk, will make the opening discussion of the Sharpe ratio and Yale's track record much clearer.
• Familiarity with standard deviation and the concept of risk-adjusted return from earlier lectures on portfolio theory will help with the Sharpe ratio discussion.

Check your understanding

1. What does the Efficient Markets Hypothesis claim, and in what sense does Shiller consider it a "half-truth"?
2. How can a fund manager manipulate a Sharpe ratio to look safer than the fund actually is, and what happened to Integral Investment Management?
3. What is the mathematical difference between a Random Walk and a first-order autoregressive process, and what does the coefficient rho control?
4. Why did Shiller's simulated Random Walk price series look so similar to the actual long-run history of the U.S. stock market?
5. According to the lecture, why might technical analysis and simple chart patterns fail to produce reliable short-run trading profits even if markets are not perfectly efficient?