sklearn.covariance.ShrunkCovariance

from sklearn.covariance import ShrunkCovariance

words from wedia:

Shrinkage estimation[edit]

If the sample size n is small and the number of considered variables p is large, the above empirical estimators of covariance and correlation are very unstable. Specifically, it is possible to furnish estimators that improve considerably upon the maximum likelihood estimate in terms of mean squared error. Moreover, for n < p, the empirical estimate of the covariance matrix becomes singular, i.e. it cannot be inverted to compute the precision matrix.

As an alternative, many methods have been suggested to improve the estimation of the covariance matrix. All of these approaches rely on the concept of shrinkage. This is implicit in Bayesian methods and in penalized maximum likelihood methods and explicit in the Stein-type shrinkage approach.

A simple version of a shrinkage estimator of the covariance matrix is constructed as follows. One considers a convex combination of the empirical estimator ({\displaystyle A}) with some suitable chosen target ({\displaystyle B}), e.g., the diagonal matrix. Subsequently, the mixing parameter ({\displaystyle \delta }) is selected to maximize the expected accuracy of the shrunken estimator. This can be done by cross-validation, or by using an analytic estimate of the shrinkage intensity. The resulting regularized estimator ({\displaystyle \delta A+(1-\delta )B}) can be shown to outperform the maximum likelihood estimator for small samples. For large samples, the shrinkage intensity will reduce to zero, hence in this case the shrinkage estimator will be identical to the empirical estimator. Apart from increased efficiency the shrinkage estimate has the additional advantage that it is always positive definite and well conditioned.

Various shrinkage targets have been proposed:

1. the identity matrix, scaled by the average sample variance;^[7][8]
2. the single-index model;^[9]
3. the constant-correlation model, where the sample variances are preserved, but all pairwise correlation coefficients are assumed to be equal to one another;^[10]
4. the two-parameter matrix, where all variances are identical, and all covariances are identical to one another (although not identical to the variances);^[11]
5. the diagonal matrix containing sample variances on the diagonal and zeros everywhere else;^[8][12]
6. the identity matrix.^[8]

A review on this topic is given, e.g., in Schäfer and Strimmer 2005.^[13] The shrinkage estimator can be generalized to a multi-target shrinkage estimator that utilizes several targets simultaneously.^[14] Software for computing a covariance shrinkage estimator is available in R (packages corpcor^[15] and ShrinkCovMat^[16]), in Python (library scikit-learn), and in MATLAB.^[17]