Expectation Maximization

• You come up with a model with some parameters $\theta$ as well as a latent variable $z$. e.g. membership in mixture model

Steps

• E: construct a lower bound. estimate latent variables $p(z|\theta)$ = ...
• M: maximize the lower bound. given $p(z|\theta)$ maximize $p(\theta|z)$ = ...
• For Gaussian Mixture: Each $X_i$ is from one of the Gaussian components, the problem is you don't know which one. Randomly assign membership to each point. Then we can estimate $p(z|\mu, \Sigma)=\frac{p(\mu,\Sigma|z)p(z)}{\sum_zp(\mu,\Sigma,z)}$, followed by $\mu$ = .., $\Sigma$ = ...