Multivariate Gaussian Distribution

Multivariate Gaussian Distribution

Ethara

This work will present a linear transformation interpretation on the multivariate Gaussian distribution. The entire derivation is originated from univariate Gaussian distributions with mean 0 and variance 1.

Recall that the density function of a univariate Gaussian distribution is given by

Therefore, when

we have

Now, suppose we haven independently identically distribution random variables x₁, …, x_n conforming to univariate Gaussian distributions with mean 0 and variance 1, then the joint density function is

By designing a vector-valued random variable,

we can re-write the joint density function as

Remarkably, here is the assumption that will lead us to a general case, that is, there exists an invertible mixing matrix B and a noise vector mu which contributes to a linear transformation from Z to X, i.e.,

Problem is, what is density function of X? Here is the derivation. (This requires some advanced linear algebra and probability theory. The relevant proof is on the horizon.)

Letting

we can obtain

where the noise vector muis the mean of X, since E[X] = E[BZ] + E[mu] = mu, and the n-by-n matrix SIGMA is a symmetric positive semi-definite covariance matrix of X, since

Moreover, since SIGMA is real symmetric, it can be factorized as

where U is a full rank orthogonal matrix containing the eigenvectors of SIGMA as its columns and LAMBDA is a diagonal matrix containing SIGMA’s eigenvalues which are non-negative. (Since SIGMA is also positive semi-definite)

Defining

to be the diagonal matrix whose entries are the square roots of the corresponding entries from LAMBDA. Noting that

we have

Therefore, the mixing matrix turns out to be

In addition, by inversing the derivation above, we can substantiate the probabilistic standardization of multivariate Gaussian distribution, i.e.,

Proof sketch:

Noting that,

where C is nonsingular, we have,

Substitute C^T(X-mu) for Y,

where

we obtain,

Incidentally, the Jacobian determinant plays an essential part in multivariate integral by substitution, geometrically suggesting the value of volume element of an n-dimensional stretched cube composed of a new basis into which the old one is linearly or nonlinearly transformed in Euclidean space.

POSTSCRIPT

Happy birthday to my dear bro, Ryan! Hopefully, one day, you will become an extraordinary mathematician like Johann Carl Friedrich Gauss.

MAY 24th, 2015