Lecture 11: Evolutionary stability: cooperation, mutation, and equilibrium

What this lecture covers

This lecture opens a new topic: what happens when strategies are not chosen by reasoning players but are hardwired, and success or failure is measured by reproduction rather than payoff maximization? It asks whether the game-theoretic tools built up over the course, dominance and Nash equilibrium, still say something useful once "choice" is replaced by "survival."

It is the first lecture in the course's short unit on evolutionary game theory, building directly on the mixed-strategy work from the previous two lectures (a strategy's frequency in a population plays a role similar to a mixing probability). After watching, you should be able to test whether a pure strategy is evolutionarily stable using the mutation-invasion argument, and explain how evolutionary stability relates to strict dominance and to Nash equilibrium.

Key ideas

• Hardwired strategies: in this model, animals don't choose strategies; genes determine behavior, and payoffs correspond to genetic fitness, so successful genes spread and unsuccessful ones die out.
• Within-species, symmetric games: the model looks at a large population randomly paired up to play a simple symmetric two-player game against members of their own species.
• Mutation-invasion test: a strategy S is evolutionarily stable if every possible small mutant strategy S', entering at a small proportion ε of the population, earns a lower average payoff than the incumbent and therefore dies out.
• Cooperation can lose: in a Prisoner's Dilemma played this way, cooperate is not evolutionarily stable (a defecting mutation invades and grows), while defect is evolutionarily stable — nature is not guaranteed to reward cooperation.
• Dominance implies instability: any strictly dominated strategy cannot be evolutionarily stable, because the dominating strategy is automatically a successful invader.
• Evolutionary stability implies Nash: if (S,S) is not a Nash equilibrium, S cannot be evolutionarily stable, because the profitable deviation doubles as a successful mutant.
• Nash does not imply evolutionary stability: a strategy can be a (weak) Nash equilibrium and still be invadable, unless it is a strict Nash equilibrium, in which case it is guaranteed to be evolutionarily stable.
• Two equivalent formal definitions: the biological definition (built from the mutation inequality with ε) is equivalent to a cleaner economic definition (Nash equilibrium, plus beating the mutant whenever there's a tie).

Walkthrough

Why study evolution alongside game theory (0:01)

The lecture explains two motivations: game theory has become a major tool in biology for analyzing animal behavior, with strategies mapped to genes and payoffs to fitness; and evolutionary ideas have fed back into economics and social science as a metaphor, for example competitive markets weeding out unprofitable firms the way natural selection weeds out unfit genes. A key difference from earlier lectures is stressed: strategies here are not chosen by reasoning individuals but are hardwired, and the model assumes simplified asexual reproduction with no gene mixing.

Testing cooperation in Prisoner's Dilemma (11:22)

Using a Prisoner's Dilemma payoff matrix and a classroom demonstration (students as cooperating "ants," a mutant group playing defect), the lecture shows that a small group of defectors earns a higher average payoff against a cooperating population and grows, so cooperate is not evolutionarily stable. Running the reverse experiment, a lone cooperative mutant entering an all-defecting population earns a lower payoff and dies out, so defect is evolutionarily stable. The lecture draws the lesson that "nature can suck" — evolutionary outcomes are not automatically efficient or cooperative, and that outcomes like real-world cooperation among lions or ants require features (such as sexual reproduction and gene sharing among relatives) missing from this simplified asexual model.

Connecting evolutionary stability to dominance and Nash equilibrium (25:46)

Generalizing from the Prisoner's Dilemma, the lecture states that a strictly dominated strategy can never be evolutionarily stable, since the dominating strategy is always a successful invader. A three-strategy symmetric game is then used to show a case where the successful invader is itself not evolutionarily stable, yet it still displaces the original strategy. This leads to the sharper claim: if (S,S) is not a Nash equilibrium, then S is not evolutionarily stable, because any strictly profitable deviation can be reinterpreted as a mutant that successfully invades.

A Nash equilibrium that is not evolutionarily stable (42:32)

A deliberately simple two-by-two game with payoffs (1,1), (0,0), (0,0), (0,0) has two Nash equilibria, (A,A) and (B,B). The lecture shows that (B,B), although Nash, is not evolutionarily stable: a mutant playing A does no worse than incumbents against other incumbents, but strictly better whenever it meets another mutant, so it grows. The flaw is that B is only a weak best response to B. The lecture concludes that a strict Nash equilibrium is guaranteed to be evolutionarily stable, closing the gap left by the counterexample.

Two formal definitions of evolutionary stability (63:00)

The lecture writes out Maynard Smith's 1972 biological definition, an inequality comparing an incumbent's payoff against a mixed population (mostly incumbents, a small fraction mutants) to a mutant's payoff against that same mixed population, required to hold for every possible mutation and for all sufficiently small mutation sizes. It then offers a cleaner, equivalent economic definition: S is evolutionarily stable if (S,S) is a Nash equilibrium, and whenever some other strategy ties as a best response, S must do strictly better against that strategy than the strategy does against itself. The lecture argues informally (leaving the full proof for the posted handout) that these two definitions coincide, splitting the argument into the case where the mutant loses badly against incumbents and the case where it loses only when meeting other mutants.

Before you watch

• Review the mixed-strategy material from the two preceding lectures in this course, since population proportions here play a role similar to mixing probabilities.
• Be comfortable identifying Nash equilibria and strictly dominated strategies in small payoff matrices, covered earlier in the course.
• Know the Prisoner's Dilemma payoff structure, introduced earlier in the course.

Check your understanding

1. Why does a defecting mutation successfully invade a population of cooperators in the Prisoner's Dilemma model, even though cooperation gives a higher payoff when two cooperators meet?
2. Explain why any strictly dominated strategy cannot be evolutionarily stable.
3. Why does (B,B) being a Nash equilibrium in the simple two-by-two example not guarantee that B is evolutionarily stable? What extra condition fixes this?
4. What is the key difference between the biological definition of evolutionary stability and the economic (Nash-based) definition, and why does the lecture consider them equivalent?
5. What assumption in this model (asexual reproduction, no gene mixing) most limits its ability to explain real cooperation seen in nature, according to the discussion?