正定与最小二乘法的联系

来源:互联网 发布:oracle mysql 优缺点 编辑:程序博客网 时间:2024/03/29 15:39

positive definite and least squares approximations

  • positive definite and least squares approximations
    • positive definite
    • ATAATA
    • least squares approximations


positive definite

When a symmetric matrix A has one of these five properties, it has them all and A is positive definite:

  1. all n eigenvalue are positive.
  2. all n principal minors(n upper left determinants) are positive.
  3. all n pivots are positive.
  4. xTAx is positive except when x=0 (this is usually the definition of positive definiteness and the energy-based definition).
  5. A equals RTR for a matrix R with independent columns.

Let us prove the fifth rule. If A=RTR, then

xTAx====xTRTRx(xTRT)Rx(Rx)TRxRx0

And the columns of R are also independent, so Rx=xTAx>0, except when x=0 and thus A is positive definite.

ATA

Am×n is almost certainly not symmetric, but ATA is square (m by m) and symmetric. We can easily get the following equations through left multiplying ATA by xT and right multiplying ATA by x:

xTATAx====xT(ATA)x(xTAT)Ax(Ax)T(Ax)Ax0

If Am×n has rank n (independent columns), then except when x=0, Ax=Ax=xT(ATA)x>0 and thus ATA is positive definite. And vice versus.

Besides, ATA is invertible only if Am×n has rank n (independent columns). To prove this, we assume Ax=0, then:

Ax(Ax)T(Ax)(xTAT)(Ax)xTAT(Ax)(ATA)x=====000xT00

From the above equations, we know solutions of Ax=0 are also solutions of (ATA)x=0. Because Am×n has a full set of column rank (independent columns), Ax=0 only has a zero solution as well as (ATA)x=0. Moreover, if ATA is invertible, then Am×n has rank n (independent columns). We also notice that if A is square and invertible, then ATA is invertible.

Overall, if all columns of Am×n are mutual independent, then (ATA) is invertible and positive definite as well, and vice versus.

least squares approximations

We have learned that least squares approximations comes from projection :

bp=eAT(bAx^)=0ATAx^=ATb

Consequently, only if ATA is invertible, then we can use linear regression to find approximate solutions x^=(ATA)1ATb to unsolvable systems of linear equations.

According to the reasonibefore, we know as long as all columns of Am×n are mutual independent, then ATA is invertible. At the same time we ought to notice that the columns of Am×n are guaranteed to be independent if they are orthoganal and even orthonormal.

In another prospective, if ATA is positive definite, then Am×n has rank n (independent columns) and thus ATA is invertible.

Overall, if ATA is positive definite or invertible, then we can find approximate solutions of least square.