# Maximum likelihood in multivariate Gaussian distribution (1)

The multivariate Gaussian distrubution is a well-known probability distribution, with a distribution function as P(x)=12πn2|Σ|12e−(x−μ)TΣ−1(x−μ)2. As we all know, if we assume the we have m sample data, the value of μ and Σ can be determined with the sample data x(1),x(2)⋯x(m) by the following equations with maximum likelihood estimation:

μΣ=x(1)+x(2)+⋯+x(m)m=∑i=1m (x(i)−μ)(x(i)−μ)T

However, how does the formula be derived?

Prerequest

To understand the points I make in this article, you have to hold the basic ideas about calculus, linear algebra, and statistics. You should be familar with derivation, matrix and maximum-likelihood estimation.

Derivation

We start out to write out the log-likelihood function of μ and Σ. We denote the likelihood function as L(μ,Σ), and

L(μ,Σ)=∏i=1m12πn2|Σ|12e−(x(i)−μ)TΣ−1(x(i)−μ)2

So, the log-likelihood function l(μ,Σ) is

l(μ,Σ)=log∏i=1m12πn2|Σ|12e−(x(i)−μ)TΣ−1(x(i)−μ)2=∑i=1mlog⎛⎝12πn2|Σ|12e−(x(i)−μ)TΣ−1(x(i)−μ)2⎞⎠=∑i=1m⎛⎝log⎛⎝12πn2|Σ|12⎞⎠−(x(i)−μ)TΣ−1(x(i)−μ)2⎞⎠=∑i=1m(−n2log(2π)−12log|Σ|−(x(i)−μ)TΣ−1(x(i)−μ)2)=−mn2log(2π)−m2log|Σ|−∑i=1m(x(i)−μ)TΣ−1(x(i)−μ)2

Then, the derivative of l with respect to μ is given by

∂l∂μ=∂(−∑mi=1(x(i)−μ)TΣ−1(x(i)−μ)2)∂μ

First, we expand the equation and get

∂l∂μ=12∂(∑mi=1((x(i))TΣ−1μ+μTΣ−1x(i)−μTΣ−1μ−(x(i))TΣ−1x(i)))∂μ

Then, we move the partial derivative into the summation

∂l∂μ=12∑i=1m∂((x(i))TΣ−1μ+μTΣ−1x(i)−μTΣ−1μ−(x(i))TΣ−1x(i))∂μ=12∑i=1m∂((x(i))TΣ−1μ+μTΣ−1x(i)−μTΣ−1μ)∂μ

Because the last term (x(i))TΣ−1x(i) hase no relation with μ

Analysis

We analyse each terms in the formula. The first point I want to make is that (x(i))TΣ−1μ=μTΣ−1x(i). Why? Let’s consider a general form of this identity which I will prove in the next few paragraphs.

Theorem: Let a, b to be n-dimentional vectors and X is a n-by-n sysmetric matrix, then the following identity holds:

aTXb=bTXa

Proof: You can expand the two terms at both sides of this identity, and you will get

aTXbbTXa=∑i=1,j=1naiXijbj=∑i=1,j=1nbiXijaj=∑i=1,j=1naiXjibj

As a result, aTXb=bTXa if and only if Xij=Xji, ∀i,j∈1,2⋯n

We can interpret the result with inner product. Recall that the inner product of the vector a and b is defined as <a,b>=aTb. And it is obvious that bTa=<b,a>=<a,b>=aTb=∑ni=1aibi. In fact, we can also define the weighted inner product of a and b as <a,b>W=∑ni=1,j=1aiWijbj=aTWb, where the matrix W is a symetric matrix called weight matrix. The term ‘weighted inner product’ is used to emphasize that we are about to “insert” a “weight” into the product of all the pairs of every items in a and b, namely, ai and bj. The inner product of two vectors a and b is cummutative, and so as its weighted counterpart with respect to a perticular weight matrix. As a result, the identity in the previous theorem, aTXb=bTXa, is now almost obvious. It is just the cummutative rule.

We continue to analyse the ∂l∂μ. Now, we can get

∂l∂μ=12∑i=1m∂(2μTΣ−1x(i)−μTΣ−1μ)∂μ

Calculate ∂l∂μ is not so easy, so we instead calculate ∂l∂μj, which is the derivative of l with respect to the j’s item in μ.

∂l∂μj=12∑i=1m(2∂μT∂μjΣ−1x(i)−∂μT∂μjΣ−1μ−μTΣ−1∂μ∂μj)

We denote the column vector with a 1 in its j’s item and 0 in its all other items as ej, it can be shown that ∂μ∂μj=ej because only the j’s item in μ has relation with μj. So we can get

∂l∂μj=12∑i=1m(2eTjΣ−1x(i)−eTjΣ−1μ−μTΣ−1ej)=12∑i=1m(2eTjΣ−1x(i)−2eTjΣ−1μ)

Note that we make use of the theorem that aTXb=bTXa if X is symetric again.

eTjΣ−1x(i)=eTj(Σ−1x(i)) is just the j’s item of Σ−1x(i) (which is simply a n-dimentional vector), so we denote it as (Σ−1x(i))j (the same as the convention of some numerical package such as numpy). As a result, the equation will be converted to

∂l∂μj=12∑i=1m(2(Σ−1x(i))j−2(Σ−1μ)j)=∑i=1m(Σ−1x(i)−Σ−1μ)j=(∑i=1m(Σ−1x(i)−Σ−1μ))j

That is, ∂l∂μj is just the j’s item of (∑mi=1(Σ−1x(i)−Σ−1μ)). As a result, ∂l∂μ is just (∑mi=1(Σ−1x(i)−Σ−1μ)).

Recall that we want to maximize l with respect to μ. So we should set ∂l∂μ to 0. Thus, we will get

∑i=1m(Σ−1x(i)−Σ−1μ)=0

The term Σ−1 is obviously non-singular. So we can get

∑i=1m(x(i)−μ)=0

and the expression for μ is thus just

μ=∑mi=1x(i)m