Maximum-Likelihood Expectation-Maximization (ML-EM)

I. Notations

X={x1,x2,...,xN} i.i.d. observed variables

Z={z1,z2,...,zN} latent variables

Θ(t)   The estimate of the parameters at iteration t

l(Θ)  The marginal log-likelihood logp(X|Θ)

II. Derivations

1. Maximum-Likelihood:

Θ̂ =argmaxΘlogp(X|Θ)=argmaxΘ∑iNlogp(xi|Θ)=argmaxΘ∑iNlog∑kKP(xi,z=k|Θ)

which is hard to compute with a gradient method.

l(Θ)=logp(X|Θ)=∑iNlog∑kKP(xi,z=k|Θ)=∑iNlog∑kKq(z=k | xi,Θ)P(xi,z=k|Θ)q(z=k | xi,Θ)

≥∑iN∑kKq(z=k | xi,Θ)logP(xi,z=k|Θ)q(z=k | xi,Θ)≡Q(q,Θ)

where q(z | x,Θ) is an arbitrary density over Z, and the inequality is given by Jessen’s inequality, i.e, Ef(x)≤f(E(x)) for convex function and Ef(x)≥f(E(x)) for concave function. Here f(x)=log(x) is a concave function.

2. Expectation-Maximization:

Thus we have the lower bound of target function l(Θ). Instead of maximizing l(Θ) directly, EM maximizes the lower-bound Q(q,Θ) via coordinate ascent:

E−step:q(t+1)=argmaxqQ(q,Θt)

M−step:Θ(t+1)=argmaxΘQ(q(t+1),Θ)

E-Step: compute q(t+1)=argmaxqQ(q,Θt) with constraint ∑Kkq(z=k | x,Θ)=1 (arbitrary density function over Z), by introducing the lagrange multiplier λ, we define

G(q)=λ(1−∑kKq(z=k | x,Θ))+∑kKq(z=k | x,Θ)logP(x,z=k|Θ)−∑kKq(z=k | x,Θ)logq(z=k | x,Θ)

∂G(q)∂q=−λ+logP(x,z=k|Θ)−logq(z=k | x,Θ)−1=0

q(z=k | x,Θ)∝P(x,z=k | Θ)=P(x,z=k|Θ)∑Kk′P(x,z=k′|Θ)=P(z=k|x,Θ)

thus, q=P(z|x,Θ) give the closest lower bound of l(Θ)

M-Step: update parameters θ, with

Θ(t+1)=argmaxΘ∑iN∑kKP(z=k | xi,Θ(t))logP(xi,z=k|Θ)P(z=k | xi,Θ(t))

=argmaxΘ∑iN∑kKP(z=k | xi,Θ(t))logP(xi,z=k|Θ)

III. Applications to Gaussian Mixture Models(GMMs)

For general mixture models, we have

P(x|Θ)=∑kKP(x,z=k|Θ)=∑kKP(z=k|Θ)P(x|z=k,Θ)

For Gaussian Mixture Models (GMMs), we have

P(x|Θ)=∑kKπk (x | μk,Σk)with∑kKπk=1

E-Step for GMMs:

Define qi,k≡P(z=k|xi,Θ), then

qi,k=P(z=k,xi,|Θ)∑Kk′P(z=k′,xi,|Θ)=πk (xi | μk,Σk)∑Kk′πk′ (xi | μk′,Σk′)

M-Step for GMMs:
update the parameters which maximizes the Log-likelihood below

Θ(t+1)=argmaxΘ∑iN∑kKqi,klogP(xi,z=k|Θ)=argmaxΘ∑iN∑kKqi,klog(πk (xi | μk,Σk))

with subject to ∑Kkπk=1, introducing the lanrange multiplier into the objective function, we thus have

G(Θ,λ)=λ(1−∑kKπk)+∑iN∑kKqi,klogπk+∑iN∑kKqi,klog(xi | μk,Σk)

∂G(Θ,λ)∂πk=0and∑kKπk=1⟹πk=∑Niqi,k∑Kk∑Niqi,k=∑Niqi,kN

∂G(Θ,λ)∂μk=0⟹μk=∑Niqi,kxi∑Niqi,k

∂G(Θ,λ)∂Σk=0⟹Σk=∑Niqi,k(xi−μk)(xi−μk)T∑Niqi,k