Seyed Masoud Hosseini · Overview · Study log · Weekly summaries · Ideas · Search · Transcript · RSS feed

Probability · Lecture 36 of 76 · 18:36

Recitation: Inferring a Discrete Variable from a Continuous Measurement

Inferring a Discrete Random Variable from a Continuous Measurement on YouTube

Study guide

What this lecture covers

This recitation problem practices the mixed version of Bayes' rule: inferring a discrete random variable X from a continuous measurement Z. X takes values 1, -1, or 0 with given probabilities, noise Y follows a two-sided exponential density, and the observed quantity is Z = X + Y. The task is to find the conditional probability that X = 1 given an observed value of Z.

It follows the lecture on continuous Bayes' rule, applying the formula for a discrete unknown measured through a continuous, noisy channel, and finishes by checking the result against several limiting cases to build confidence that the formula is correct.

Key ideas

  • Mixed Bayes' rule: for a discrete X and continuous Z, the posterior PMF of X given Z = z is the prior PMF of X times the conditional density of Z given X, divided by the total probability, obtained via the law of total probability over the discrete values of X.
  • Shifting a density to condition on X: because Z = X + Y, the conditional density of Z given X = 1 is just the noise density shifted by 1, and given X = -1 it is shifted by -1.
  • Sanity-checking with limiting cases: plugging in extreme values of the parameters (P near 0 or 1, lambda near 0 or infinity) into the derived formula should match intuitive expectations, which confirms the algebra.
  • Uninformative measurements: when the noise is very spread out (lambda near 0), observing Z does not change the belief about X, so the posterior equals the prior.
  • Highly informative measurements: when the noise is nearly zero (lambda to infinity), Z essentially equals X, so the sign of the observed z determines X with near certainty.

Walkthrough

Setting up the model (0:00)

X is discrete with probability P of being 1, 1-P of being -1, and 0 otherwise. Y is a continuous two-sided exponential noise term with rate lambda. Z = X + Y is the observed, continuous measurement, and the goal is P(X=1 | Z=z).

Applying the mixed Bayes' rule (1:02)

The formula multiplies the prior PMF of X by the conditional density of Z given X, then divides by the total probability of Z=z, expanded using the law of total probability over X = 1 and X = -1. Since Z given X=1 is just the noise density shifted right by 1, and Z given X=-1 is the noise density shifted left by 1, both conditional densities can be written explicitly and substituted in.

Simplifying the expression (5:15)

After substituting the shifted exponential densities, the common factors cancel, leaving a formula for the posterior probability in terms of P, lambda, and z, involving a ratio of exponential terms.

Checking limiting cases (7:21)

As P goes to 0, the posterior probability correctly goes to 0, since X can never be 1. As P goes to 1, the posterior goes to 1. As lambda goes to 0 (very spread-out noise), the posterior simplifies back to P, meaning the measurement adds no information. As lambda goes to infinity (nearly noiseless), the posterior approaches 1 when z is positive and 0 when z is negative, since Z essentially reveals X directly.

Before you watch

  • Review the discrete-and-continuous mixed form of Bayes' rule from the lecture on continuous Bayes' rule and derived distributions.
  • Be familiar with the two-sided (Laplace-style) exponential density and how shifting a random variable shifts its density.
  • Knowing the law of total probability for a discrete variable helps follow how the denominator is built.

Check your understanding

  1. Why is the conditional density of Z given X=1 just the noise density shifted by 1?
  2. What role does the law of total probability play in the denominator of the mixed Bayes' rule formula?
  3. Why does the posterior probability equal the prior P when lambda goes to 0?
  4. Why does observing the sign of z determine X almost exactly when lambda goes to infinity?

Vocabulary

posterior PMF (phrase)
The updated probability table for a discrete unknown, after seeing evidence.
We compute the posterior PMF of X given the measurement Z.
two-sided exponential (phrase)
A distribution shaped like two mirrored exponential curves meeting at zero.
The noise follows a two-sided exponential density.
shift (verb)
To move a distribution to a different position without changing its shape.
Adding X shifts the noise density by 1.
sanity-check (verb)
To test a result against simple cases to see if it makes sense.
We sanity-check the formula using extreme parameter values.
limiting case (phrase)
A special, extreme version of a situation used to test a general result.
The limiting case where lambda is near infinity is very informative.
uninformative (adjective)
Giving no useful information.
A very noisy measurement is uninformative about X.
informative (adjective)
Giving useful information that changes belief.
A nearly noiseless measurement is highly informative.
ratio (noun)
The result of dividing one quantity by another.
The final formula involves a ratio of exponential terms.
cancel out (phrasal verb)
To remove each other so nothing is left.
The common factors cancel out in the simplification.
extreme value (phrase)
A value at or near the very edge of what is possible.
We test extreme values of P and lambda to check the formula.
Bayes' rule (phrase)
A formula for updating a belief about an unknown after seeing evidence.
Bayes' rule combines the prior with the observed measurement.
prior (noun)
The belief about an unknown before seeing any evidence.
P is the prior probability that X equals 1.
noise (noun)
Random, unwanted variation added to a signal or measurement.
Y represents the noise added to X to produce the measurement Z.
channel (noun)
A path through which a signal or measurement is transmitted, possibly with noise.
X is measured through a continuous, noisy channel.
algebra (noun)
The use of symbols and equations to represent and solve mathematical relationships.
Checking limiting cases confirms the algebra is correct.
intuitive (adjective)
Matching what you would naturally expect, without needing formal proof.
The limiting cases match intuitive expectations about the formula.
near certainty (phrase)
A probability extremely close to 1, almost guaranteed.
A nearly noiseless measurement determines X with near certainty.
exponential density (phrase)
A probability density that decays smoothly at a constant relative rate.
The noise follows a two-sided exponential density with rate lambda.
discrete unknown (phrase)
An unknown quantity that can only take separate, distinct values.
The mixed Bayes' rule formula applies to a discrete unknown measured continuously.
plug in (phrasal verb)
To insert specific values into a formula.
We plug in extreme values of P and lambda to test the result.
substitute (verb)
To put one expression in place of another in a formula.
The shifted densities are substituted into the Bayes' rule formula.
explicit (adjective)
Stated clearly and in full detail.
Both conditional densities can be written explicitly.
correctly (adverb)
In a way that matches the true or expected result.
The posterior correctly goes to 0 as P goes to 0.
approach (verb)
To get closer and closer to a value without necessarily reaching it exactly.
The posterior approaches 1 as lambda goes to infinity.
denominator (noun)
The bottom number in a fraction.
The total probability of Z=z forms the denominator of Bayes' rule.

Chapters

From the YouTube description

MIT 6.041SC Probabilistic Systems Analysis and Applied Probability, Fall 2013
View the complete course: http://ocw.mit.edu/6-041SCF13
Instructor: Jimmy Li

License: Creative Commons BY-NC-SA
More information at http://ocw.mit.edu/terms
More courses at http://ocw.mit.edu

← Lecture 10: Continuous Bayes' Rule; Derived Distributions · Inferring a Continuous Random Variable from a Discrete Measurement →