scikit-learn（工程中用的相对较多的模型介绍）：1.13. Feature selection

参考：http://scikit-learn.org/stable/modules/feature_selection.html

The classes in the sklearn.feature_selection module can be used for feature selection/dimensionality reduction on sample sets, either to improve estimators’ accuracy scores or to boost their performance on very high-dimensional datasets.

1、removing features with low variance

VarianceThreshold 是特征选择的简单baseline方法，他删除方差达不到阈值的特征。默认情况下，删除all zero-variance features, i.e. features that have the same value in all samples.

假设我们想要删除  超过80%的样本数都是0或都是1（假设是boolean features）  的所有特征，由于boolean features是bernoulli随机变量，所以方差为Var[X] = p(1-p)，所以我们可以使用阈值0.8*（1-0.8）：

>>> from sklearn.feature_selection import VarianceThreshold>>> X = [[0, 0, 1], [0, 1, 0], [1, 0, 0], [0, 1, 1], [0, 1, 0], [0, 1, 1]]>>> sel = VarianceThreshold(threshold=(.8 * (1 - .8)))>>> sel.fit_transform(X)array([[0, 1],       [1, 0],       [0, 0],       [1, 1],       [1, 0],       [1, 1]])

删除了第一列，因为p=5/6 > 0.8。

2、Univariate feature selection（单变量特征选择）（我用这个非常多）

Univariate feature selection基于univariate statistical tests（单变量统计检验），分为：

• SelectKBest removes all but the  highest scoring features

• SelectPercentile removes all but a user-specified highest scoring percentage of features

• using common univariate statistical tests for each feature: false positive rate SelectFpr, false discovery rate SelectFdr, or family wise error SelectFwe.

• GenericUnivariateSelect allows to perform univariate feature selection with a configurable strategy. This allows to select the best univariate selection strategy with hyper-parameter search estimator.

例如，我们可以对样本集使用卡方检测，进而选择最好的两个features：

>>> from sklearn.datasets import load_iris>>> from sklearn.feature_selection import SelectKBest>>> from sklearn.feature_selection import chi2>>> iris = load_iris()>>> X, y = iris.data, iris.target>>> X.shape(150, 4)>>> X_new = SelectKBest(chi2, k=2).fit_transform(X, y)>>> X_new.shape(150, 2)

几个注意点：

1）These objects take as input a scoring function that returns univariate p-values：For regression: f_regression；For classification: chi2 or f_classif。 

Beware not to use a regression scoring function with a classification problem, you will get useless results.

2）Feature selection with sparse data：If you use sparse data (i.e. data represented as sparse matrices), only chi2 will deal with the data without making it dense.

例子：Univariate Feature Selection

3、recursive feature elimination（递归特征消除）

Given an external estimator that assigns weights to features (e.g., the coefficients of a linear model), recursive feature elimination (RFE) is to select features by recursively considering smaller and smaller sets of features.（从最初的所有特征集到逐步删除一个feature< features whose absolute weights are the smallest are pruned from the current set features>，最后达到满足条件的features个数）。

RFECV performs RFE in a cross-validation loop to find the optimal number of features.：

• Recursive feature elimination: A recursive feature elimination example showing the relevance of pixels in a digit classification task.
• Recursive feature elimination with cross-validation: A recursive feature elimination example with automatic tuning of the number of features selected with cross-validation.

4、L1-based feature selection

L1的sparse作用就不说了：

>>> from sklearn.svm import LinearSVC>>> from sklearn.datasets import load_iris>>> iris = load_iris()>>> X, y = iris.data, iris.target>>> X.shape(150, 4)>>> X_new = LinearSVC(C=0.01, penalty="l1", dual=False).fit_transform(X, y)>>> X_new.shape(150, 3)

With SVMs and logistic-regression, the parameter C controls the sparsity: the smaller C the fewer features selected. With Lasso, the higher the alpha parameter, the fewer features selected.

Examples:

• Classification of text documents using sparse features: Comparison of different algorithms for document classification including L1-based feature selection.

5、Tree-based features selection（这个也用的比价多）

Tree-based estimators (see the sklearn.tree module and forest of trees in the sklearn.ensemble module) can be used to compute feature importances, which in turn can be used to discard irrelevant features:

>>>

>>> from sklearn.ensemble import ExtraTreesClassifier>>> from sklearn.datasets import load_iris>>> iris = load_iris()>>> X, y = iris.data, iris.target>>> X.shape(150, 4)>>> clf = ExtraTreesClassifier()>>> X_new = clf.fit(X, y).transform(X)>>> clf.feature_importances_  array([ 0.04...,  0.05...,  0.4...,  0.4...])>>> X_new.shape               (150, 2)

• Feature importances with forests of trees: example on synthetic data showing the recovery of the actually meaningful features.
• Pixel importances with a parallel forest of trees: example on face recognition data.

6、Feature selection as part of a pipeline

Feature selection is usually used as a pre-processing step before doing the actual learning. The recommended way to do this in scikit-learn is to use asklearn.pipeline.Pipeline:

clf = Pipeline([  ('feature_selection', LinearSVC(penalty="l1")),  ('classification', RandomForestClassifier())])clf.fit(X, y)

In this snippet we make use of a sklearn.svm.LinearSVC to evaluate feature importances and select the most relevant features. Then, asklearn.ensemble.RandomForestClassifier is trained on the transformed output, i.e. using only relevant features. You can perform similar operations with the other feature selection methods and also classifiers that provide a way to evaluate feature importances of course. See the sklearn.pipeline.Pipelineexamples for more details.