1.11. 集成方法
来源:互联网 发布:java 搜索框架 编辑:程序博客网 时间:2024/05/16 10:47
1.11. 集成方法
集成方法结合不同分类器的预测结果, 这些分类器分别来自于不同的学习算法, 相比于单一分类器以提高分类器的泛化/健壮性。
集成方法通常分为两类:
在 一般方法 中,方法的原理是使用若干个独立的分类器, 然后取这若干个分类器的平均结果作为集合方法结果。 一般情况下,集成分类器(the combined estimator)通常优于它包含的单个 分类器的效果,因为它的方差更小。
Examples: Bagging methods, Forests of randomized trees, ...
相比之下,在 boosting 方法中,基础分类器(base estimators)按顺序创建 然后试图减少合成分类器的bias。依据是将若干个弱模型结合产生强集成(分类器)。
Examples: AdaBoost, Gradient Tree Boosting, ...
1.11.1. Bagging meta-estimator
在集成算法中,bagging方法形成一个新的算法, 它基于一些原训练集的子集训练出的若干分类器。 然后把这些分类器的预测结果组合成最终预测结果。 这些方法通常用来减小基分类器的方差(例如决策树), 通过把construction procedure随机化,然后在外部进行集成。 在大部分情况下,bagging是一个非常简单, 相对于单一模型提升效果的方法,并且跟基算法无关。 由于bagging也是一种能减少过拟合的算法, 所以它能在复杂模型的情况下很有效(e.g., fully developed decision trees), 例如boosting方法通常只能在弱模型下效果比较好(e.g.,shallow decision trees)。
Bagging方法有很多种,但不同的地方主要在于 他们从训练集中得到随机子集的方式不同。
- 当随机数据集为样本的随机子集,然后该算法被称为Pasting [B1999]。
- 当样本的选取为有放回抽样时,该算法称为Bagging [B1996]。
- 数据集的随机子集为特征的随机子集,然后该方法被称为随机子空间[H1998]_。
- 最后,当基分类器是根据特征和样本构建的随机子集, 该方法称为Random Patches [LG2012]。
In scikit-learn, bagging methods are offered as a unified BaggingClassifier
meta-estimator (resp. BaggingRegressor
), taking as input a user-specified base estimator along with parameters specifying the strategy to draw random subsets. In particular, max_samples
and max_features
control the size of the subsets (in terms of samples and features), while bootstrap
and bootstrap_features
control whether samples and features are drawn with or without replacement. When using a subset of the available samples the generalization error can be estimated with the out-of-bag samples by setting oob_score=True
. As an example, the snippet below illustrates how to instantiate a bagging ensemble of KNeighborsClassifier
base estimators, each built on random subsets of 50% of the samples and 50% of the features.
在scikit-learn中,Bagging方法提供一个统一的baggingclassifier元估计(或。baggingregressor),以作为输入用户指定的基数估计与参数指定要绘制的任意子集的策略。特别是,max_samples和max_features控制集合的大小(在样本和特征),而bootstrap和bootstrap_features控制是否样品和功能绘制或不更换。利用整体误差的估计可以与出袋样品通过设置oob_score =True可用的样本子集时。例如,下面的代码片段说明了如何实例化一个词袋模型kneighborsclassifier基础估计,样本随机选取50%的特征和对应的50%的label。
Examples:
- Single estimator versus bagging: bias-variance decomposition
References
1.11.2. Forests of randomized trees
The sklearn.ensemble
module includes two averaging algorithms based on randomized decision trees: the RandomForest algorithm and the Extra-Trees method. Both algorithms are perturb-and-combine techniques [B1998] specifically designed for trees. This means a diverse set of classifiers is created by introducing randomness in the classifier construction. The prediction of the ensemble is given as the averaged prediction of the individual classifiers.
As other classifiers, forest classifiers have to be fitted with two arrays: a sparse or dense array X of size [n_samples, n_features]
holding the training samples, and an array Y of size [n_samples]
holding the target values (class labels) for the training samples:
Like decision trees, forests of trees also extend to multi-output problems (if Y is an array of size [n_samples, n_outputs]
).
1.11.2.1. Random Forests
In random forests (see RandomForestClassifier
and RandomForestRegressor
classes), each tree in the ensemble is built from a sample drawn with replacement (i.e., a bootstrap sample) from the training set. In addition, when splitting a node during the construction of the tree, the split that is chosen is no longer the best split among all features. Instead, the split that is picked is the best split among a random subset of the features. As a result of this randomness, the bias of the forest usually slightly increases (with respect to the bias of a single non-random tree) but, due to averaging, its variance also decreases, usually more than compensating for the increase in bias, hence yielding an overall better model.
随机森林(见RandomForestClassifier和randomforestregressor类),每个树是从所有训练集的一个样本得出置换。此外,在树构建期间分割节点时,所选择的分割不再是所有特征之间最好的分割。相反,被选中的分割是特征的随机子集之间最好的分割。由于这种随机性,森林的偏差通常会稍微增加(相对于单个非随机树的偏差),但由于平均值,其方差也减小,通常比偏置增加补偿更多,从而产生一个整体更好的模型。
In contrast to the original publication [B2001], the scikit-learn implementation combines classifiers by averaging their probabilistic prediction, instead of letting each classifier vote for a single class.
1.11.2.2. Extremely Randomized Trees 极端随机森林
In extremely randomized trees (see ExtraTreesClassifier
and ExtraTreesRegressor
classes), randomness goes one step further in the way splits are computed. As in random forests, a random subset of candidate features is used, but instead of looking for the most discriminative thresholds, thresholds are drawn at random for each candidate feature and the best of these randomly-generated thresholds is picked as the splitting rule. This usually allows to reduce the variance of the model a bit more, at the expense of a slightly greater increase in bias:
在极端随机森林中,在计算分割方法随机性更进了一步。在随机森林中,使用候选特征的随机子集,而不是寻找最具关键的阈值,随机抽取每个候选特征的阈值,并将这些随机生成的阈值作为划分规则选出。这通常允许稍微减少模型的方差,而牺牲稍微大幅度的偏差:
1.11.2.3. Parameters
The main parameters to adjust when using these methods is n_estimators
and max_features
. The former is the number of trees in the forest. The larger the better, but also the longer it will take to compute. In addition, note that results will stop getting significantly better beyond a critical number of trees. The latter is the size of the random subsets of features to consider when splitting a node. The lower the greater the reduction of variance, but also the greater the increase in bias. Empirical good default values are max_features=n_features
for regression problems, and max_features=sqrt(n_features)
for classification tasks (where n_features
is the number of features in the data). Good results are often achieved when setting max_depth=None
in combination with min_samples_split=1
(i.e., when fully developing the trees). Bear in mind though that these values are usually not optimal, and might result in models that consume a lot of ram. The best parameter values should always be cross-validated. In addition, note that in random forests, bootstrap samples are used by default (bootstrap=True
) while the default strategy for extra-trees is to use the whole dataset (bootstrap=False
). When using bootstrap sampling the generalization error can be estimated on the left out or out-of-bag samples. This can be enabled by setting oob_score=True
.
1.11.2.4. Parallelization
Finally, this module also features the parallel construction of the trees and the parallel computation of the predictions through the n_jobs
parameter. If n_jobs=k
then computations are partitioned into k
jobs, and run on k
cores of the machine. If n_jobs=-1
then all cores available on the machine are used. Note that because of inter-process communication overhead, the speedup might not be linear (i.e., using k
jobs will unfortunately not be k
times as fast). Significant speedup can still be achieved though when building a large number of trees, or when building a single tree requires a fair amount of time (e.g., on large datasets).
Examples:
- Plot the decision surfaces of ensembles of trees on the iris dataset
- Pixel importances with a parallel forest of trees
- Face completion with a multi-output estimators
References
- Breiman, “Random Forests”, Machine Learning, 45(1), 5-32, 2001.
- Breiman, “Arcing Classifiers”, Annals of Statistics 1998.
1.11.2.5. Feature importance evaluation
The relative rank (i.e. depth) of a feature used as a decision node in a tree can be used to assess the relative importance of that feature with respect to the predictability of the target variable. Features used at the top of the tree are used contribute to the final prediction decision of a larger fraction of the input samples. The expected fraction of the samples they contribute to can thus be used as an estimate of the relative importance of the features.
By averaging those expected activity rates over several randomized trees one can reduce the variance of such an estimate and use it for feature selection.
The following example shows a color-coded representation of the relative importances of each individual pixel for a face recognition task using a ExtraTreesClassifier
model.
In practice those estimates are stored as an attribute named feature_importances_
on the fitted model. This is an array with shape (n_features,)
whose values are positive and sum to 1.0. The higher the value, the more important is the contribution of the matching feature to the prediction function.
Examples:
- Pixel importances with a parallel forest of trees
- Feature importances with forests of trees
1.11.2.6. Totally Random Trees Embedding
RandomTreesEmbedding
implements an unsupervised transformation of the data. Using a forest of completely random trees, RandomTreesEmbedding
encodes the data by the indices of the leaves a data point ends up in. This index is then encoded in a one-of-K manner, leading to a high dimensional, sparse binary coding. This coding can be computed very efficoiently and can then be used as a basis fr other learning tasks. The size and sparsity of the code can be influenced by choosing the number of trees and the maximum depth per tree. For each tree in the ensemble, the coding contains one entry of one. The size of the coding is at most n_estimators * 2 ** max_depth
, the maximum number of leaves in the forest.
As neighboring data points are more likely to lie within the same leaf of a tree, the transformation performs an implicit, non-parametric density estimation.
Examples:
- Hashing feature transformation using Totally Random Trees
- Manifold learning on handwritten digits: Locally Linear Embedding, Isomap... compares non-linear dimensionality reduction techniques on handwritten digits.
- Feature transformations with ensembles of trees compares supervised and unsupervised tree based feature transformations.
See also
Manifold learning techniques can also be useful to derive non-linear representations of feature space, also these approaches focus also on dimensionality reduction.
1.11.3. AdaBoost
The module sklearn.ensemble
includes the popular boosting algorithm AdaBoost, introduced in 1995 by Freund and Schapire [FS1995].
The core principle of AdaBoost is to fit a sequence of weak learners (i.e., models that are only slightly better than random guessing, such as small decision trees) on repeatedly modified versions of the data. The predictions from all of them are then combined through a weighted majority vote (or sum) to produce the final prediction. The data modifications at each so-called boosting iteration consist of applying weights , , ..., to each of the training samples. Initially, those weights are all set to , so that the first step simply trains a weak learner on the original data. For each successive iteration, the sample weights are individually modified and the learning algorithm is reapplied to the reweighted data. At a given step, those training examples that were incorrectly predicted by the boosted model induced at the previous step have their weights increased, whereas the weights are decreased for those that were predicted correctly. As iterations proceed, examples that are difficult to predict receive ever-increasing influence. Each subsequent weak learner is thereby forced to concentrate on the examples that are missed by the previous ones in the sequence [HTF].
AdaBoost的核心原则是将一系列弱学习机(即只比随机猜测稍微好一点的模型,比如小型决策树)放在反复迭代,不断修改参数的数据版本中。然后通过加权投票(或总数)来预测所有的预测结果。修改数据在每一个所谓的提高迭代包括应用权重w_1,w_2,…,w_n每个训练样本。最初,这些权重都将w_i = 1 / N,所以,第一步就是训练弱分类器对原始数据。每个连续的迭代,样本权重分别修改和学习算法重新的加权数据。在给定的步骤中,在前一步引起的增强模型错误地预测了那些训练实例,它们的权重增加,而正确预测的权重则降低。随着迭代的进行,难以预测的例子受到越来越多的影响。每个后续的弱学习从而不得不集中精力,在序列[HTF]以往错过的例子。
AdaBoost can be used both for classification and regression problems:
- For multi-class classification,
AdaBoostClassifier
implements AdaBoost-SAMME and AdaBoost-SAMME.R [ZZRH2009].- For regression,
AdaBoostRegressor
implements AdaBoost.R2 [D1997].
1.11.3.1. Usage
The following example shows how to fit an AdaBoost classifier with 100 weak learners:
The number of weak learners is controlled by the parameter n_estimators
. The learning_rate
parameter controls the contribution of the weak learners in the final combination. By default, weak learners are decision stumps. Different weak learners can be specified through the base_estimator
parameter. The main parameters to tune to obtain good results are n_estimators
and the complexity of the base estimators (e.g., its depth max_depth
or minimum required number of samples at a leaf min_samples_leaf
in case of decision trees).
Examples:
- Discrete versus Real AdaBoost compares the classification error of a decision stump, decision tree, and a boosted decision stump using AdaBoost-SAMME and AdaBoost-SAMME.R.
- Multi-class AdaBoosted Decision Trees shows the performance of AdaBoost-SAMME and AdaBoost-SAMME.R on a multi-class problem.
- Two-class AdaBoost shows the decision boundary and decision function values for a non-linearly separable two-class problem using AdaBoost-SAMME.
- Decision Tree Regression with AdaBoost demonstrates regression with the AdaBoost.R2 algorithm.
References
- Drucker. “Improving Regressors using Boosting Techniques”, 1997.
1.11.4. Gradient Tree Boosting
Gradient Tree Boosting or Gradient Boosted Regression Trees (GBRT) is a generalization of boosting to arbitrary differentiable loss functions. GBRT is an accurate and effective off-the-shelf procedure that can be used for both regression and classification problems. Gradient Tree Boosting models are used in a variety of areas including Web search ranking and ecology.
The advantages of GBRT are:
- Natural handling of data of mixed type (= heterogeneous features)
- Predictive power
- Robustness to outliers in output space (via robust loss functions)
The disadvantages of GBRT are:
- Scalability, due to the sequential nature of boosting it can hardly be parallelized.
The module sklearn.ensemble
provides methods for both classification and regression via gradient boosted regression trees.
1.11.4.1. Classification
GradientBoostingClassifier
supports both binary and multi-class classification. The following example shows how to fit a gradient boosting classifier with 100 decision stumps as weak learners:
The number of weak learners (i.e. regression trees) is controlled by the parameter n_estimators
; The size of each tree can be controlled either by setting the tree depth via max_depth
or by setting the number of leaf nodes via max_leaf_nodes
. The learning_rate
is a hyper-parameter in the range (0.0, 1.0] that controls overfitting via shrinkage .
Note
Classification with more than 2 classes requires the induction of n_classes
regression trees at each at each iteration, thus, the total number of induced trees equals n_classes * n_estimators
. For datasets with a large number of classes we strongly recommend to use RandomForestClassifier
as an alternative to GradientBoostingClassifier
.
1.11.4.2. Regression
GradientBoostingRegressor
supports a number of different loss functions for regression which can be specified via the argument loss
; the default loss function for regression is least squares ('ls'
).
The figure below shows the results of applying GradientBoostingRegressor
with least squares loss and 500 base learners to the Boston house price dataset (sklearn.datasets.load_boston
). The plot on the left shows the train and test error at each iteration. The train error at each iteration is stored in thetrain_score_
attribute of the gradient boosting model. The test error at each iterations can be obtained via the staged_predict
method which returns a generator that yields the predictions at each stage. Plots like these can be used to determine the optimal number of trees (i.e. n_estimators
) by early stopping. The plot on the right shows the feature importances which can be obtained via the feature_importances_
property.
Examples:
- Gradient Boosting regression
- Gradient Boosting Out-of-Bag estimates
1.11.4.3. Fitting additional weak-learners
Both GradientBoostingRegressor
and GradientBoostingClassifier
support warm_start=True
which allows you to add more estimators to an already fitted model.
1.11.4.4. Controlling the tree size
The size of the regression tree base learners defines the level of variable interactions that can be captured by the gradient boosting model. In general, a tree of depth h
can capture interactions of order h
. There are two ways in which the size of the individual regression trees can be controlled.
If you specify max_depth=h
then complete binary trees of depth h
will be grown. Such trees will have (at most) 2**h
leaf nodes and 2**h - 1
split nodes.
Alternatively, you can control the tree size by specifying the number of leaf nodes via the parameter max_leaf_nodes
. In this case, trees will be grown using best-first search where nodes with the highest improvement in impurity will be expanded first. A tree with max_leaf_nodes=k
has k - 1
split nodes and thus can model interactions of up to order max_leaf_nodes - 1
.
We found that max_leaf_nodes=k
gives comparable results to max_depth=k-1
but is significantly faster to train at the expense of a slightly higher training error. The parameter max_leaf_nodes
corresponds to the variable J
in the chapter on gradient boosting in [F2001] and is related to the parameterinteraction.depth
in R’s gbm package where max_leaf_nodes == interaction.depth + 1
.
1.11.4.5. Mathematical formulation
GBRT considers additive models of the following form:
where are the basis functions which are usually called weak learners in the context of boosting. Gradient Tree Boosting uses decision trees of fixed size as weak learners. Decision trees have a number of abilities that make them valuable for boosting, namely the ability to handle data of mixed type and the ability to model complex functions.
Similar to other boosting algorithms GBRT builds the additive model in a forward stagewise fashion:
At each stage the decision tree is chosen to minimize the loss function given the current model and its fit
The initial model is problem specific, for least-squares regression one usually chooses the mean of the target values.
Note
The initial model can also be specified via the init
argument. The passed object has to implement fit
and predict
.
Gradient Boosting attempts to solve this minimization problem numerically via steepest descent: The steepest descent direction is the negative gradient of the loss function evaluated at the current model which can be calculated for any differentiable loss function:
Where the step length is chosen using line search:
The algorithms for regression and classification only differ in the concrete loss function used.
1.11.4.5.1. Loss Functions
The following loss functions are supported and can be specified using the parameter loss
:
- Regression
- Least squares (
'ls'
): The natural choice for regression due to its superior computational properties. The initial model is given by the mean of the target values.- Least absolute deviation (
'lad'
): A robust loss function for regression. The initial model is given by the median of the target values.- Huber (
'huber'
): Another robust loss function that combines least squares and least absolute deviation; usealpha
to control the sensitivity with regards to outliers (see [F2001] for more details).- Quantile (
'quantile'
): A loss function for quantile regression. Use0 < alpha < 1
to specify the quantile. This loss function can be used to create prediction intervals (see Prediction Intervals for Gradient Boosting Regression).- Classification
- Binomial deviance (
'deviance'
): The negative binomial log-likelihood loss function for binary classification (provides probability estimates). The initial model is given by the log odds-ratio.- Multinomial deviance (
'deviance'
): The negative multinomial log-likelihood loss function for multi-class classification withn_classes
mutually exclusive classes. It provides probability estimates. The initial model is given by the prior probability of each class. At each iterationn_classes
regression trees have to be constructed which makes GBRT rather inefficient for data sets with a large number of classes.- Exponential loss (
'exponential'
): The same loss function asAdaBoostClassifier
. Less robust to mislabeled examples than'deviance'
; can only be used for binary classification.
1.11.4.6. Regularization
1.11.4.6.1. Shrinkage
[F2001] proposed a simple regularization strategy that scales the contribution of each weak learner by a factor :
The parameter is also called the learning rate because it scales the step length the the gradient descent procedure; it can be set via the learning_rate
parameter.
The parameter learning_rate
strongly interacts with the parameter n_estimators
, the number of weak learners to fit. Smaller values of learning_rate
require larger numbers of weak learners to maintain a constant training error. Empirical evidence suggests that small values of learning_rate
favor better test error. [HTF2009] recommend to set the learning rate to a small constant (e.g. learning_rate <= 0.1
) and choose n_estimators
by early stopping. For a more detailed discussion of the interaction between learning_rate
and n_estimators
see [R2007].
1.11.4.6.2. Subsampling
[F1999] proposed stochastic gradient boosting, which combines gradient boosting with bootstrap averaging (bagging). At each iteration the base classifier is trained on a fraction subsample
of the available training data. The subsample is drawn without replacement. A typical value of subsample
is 0.5.
The figure below illustrates the effect of shrinkage and subsampling on the goodness-of-fit of the model. We can clearly see that shrinkage outperforms no-shrinkage. Subsampling with shrinkage can further increase the accuracy of the model. Subsampling without shrinkage, on the other hand, does poorly.
Another strategy to reduce the variance is by subsampling the features analogous to the random splits in RandomForestClassifier
. The number of subsampled features can be controlled via the max_features
parameter.
Note
Using a small max_features
value can significantly decrease the runtime.
Stochastic gradient boosting allows to compute out-of-bag estimates of the test deviance by computing the improvement in deviance on the examples that are not included in the bootstrap sample (i.e. the out-of-bag examples). The improvements are stored in the attribute oob_improvement_
. oob_improvement_[i]
holds the improvement in terms of the loss on the OOB samples if you add the i-th stage to the current predictions. Out-of-bag estimates can be used for model selection, for example to determine the optimal number of iterations. OOB estimates are usually very pessimistic thus we recommend to use cross-validation instead and only use OOB if cross-validation is too time consuming.
Examples:
- Gradient Boosting regularization
- Gradient Boosting Out-of-Bag estimates
- OOB Errors for Random Forests
1.11.4.7. Interpretation
Individual decision trees can be interpreted easily by simply visualizing the tree structure. Gradient boosting models, however, comprise hundreds of regression trees thus they cannot be easily interpreted by visual inspection of the individual trees. Fortunately, a number of techniques have been proposed to summarize and interpret gradient boosting models.
1.11.4.7.1. Feature importance
Often features do not contribute equally to predict the target response; in many situations the majority of the features are in fact irrelevant. When interpreting a model, the first question usually is: what are those important features and how do they contributing in predicting the target response?
Individual decision trees intrinsically perform feature selection by selecting appropriate split points. This information can be used to measure the importance of each feature; the basic idea is: the more often a feature is used in the split points of a tree the more important that feature is. This notion of importance can be extended to decision tree ensembles by simply averaging the feature importance of each tree (see Feature importance evaluation for more details).
The feature importance scores of a fit gradient boosting model can be accessed via the feature_importances_
property:
Examples:
- Gradient Boosting regression
1.11.4.7.2. Partial dependence
Partial dependence plots (PDP) show the dependence between the target response and a set of ‘target’ features, marginalizing over the values of all other features (the ‘complement’ features). Intuitively, we can interpret the partial dependence as the expected target response [1] as a function of the ‘target’ features [2].
Due to the limits of human perception the size of the target feature set must be small (usually, one or two) thus the target features are usually chosen among the most important features.
The Figure below shows four one-way and one two-way partial dependence plots for the California housing dataset:
One-way PDPs tell us about the interaction between the target response and the target feature (e.g. linear, non-linear). The upper left plot in the above Figure shows the effect of the median income in a district on the median house price; we can clearly see a linear relationship among them.
PDPs with two target features show the interactions among the two features. For example, the two-variable PDP in the above Figure shows the dependence of median house price on joint values of house age and avg. occupants per household. We can clearly see an interaction between the two features: For an avg. occupancy greater than two, the house price is nearly independent of the house age, whereas for values less than two there is a strong dependence on age.
The module partial_dependence
provides a convenience function plot_partial_dependence
to create one-way and two-way partial dependence plots. In the below example we show how to create a grid of partial dependence plots: two one-way PDPs for the features 0
and 1
and a two-way PDP between the two features:
For multi-class models, you need to set the class label for which the PDPs should be created via the label
argument:
If you need the raw values of the partial dependence function rather than the plots you can use the partial_dependence
function:
The function requires either the argument grid
which specifies the values of the target features on which the partial dependence function should be evaluated or the argument X
which is a convenience mode for automatically creating grid
from the training data. If X
is given, the axes
value returned by the function gives the axis for each target feature.
For each value of the ‘target’ features in the grid
the partial dependence function need to marginalize the predictions of a tree over all possible values of the ‘complement’ features. In decision trees this function can be evaluated efficiently without reference to the training data. For each grid point a weighted tree traversal is performed: if a split node involves a ‘target’ feature, the corresponding left or right branch is followed, otherwise both branches are followed, each branch is weighted by the fraction of training samples that entered that branch. Finally, the partial dependence is given by a weighted average of all visited leaves. For tree ensembles the results of each individual tree are again averaged.
Footnotes
loss='deviance'
the target response is logit(p).init
model.Examples:
- Partial Dependence Plots
References
- Friedman, “Stochastic Gradient Boosting”, 1999
- Hastie, R. Tibshirani and J. Friedman, “Elements of Statistical Learning Ed. 2”, Springer, 2009.
- Ridgeway, “Generalized Boosted Models: A guide to the gbm package”, 2007
1.11.5. VotingClassifier
The idea behind the voting classifier implementation is to combine conceptually different machine learning classifiers and use a majority vote or the average predicted probabilities (soft vote) to predict the class labels. Such a classifier can be useful for a set of equally well performing model in order to balance out their individual weaknesses.
1.11.5.1. Majority Class Labels (Majority/Hard Voting)
In majority voting, the predicted class label for a particular sample is the class label that represents the majority (mode) of the class labels predicted by each individual classifier.
E.g., if the prediction for a given sample is
- classifier 1 -> class 1
- classifier 2 -> class 1
- classifier 3 -> class 2
the VotingClassifier (with voting='hard'
) would classify the sample as “class 1” based on the majority class label.
In the cases of a tie, the VotingClassifier will select the class based on the ascending sort order. E.g., in the following scenario
- classifier 1 -> class 2
- classifier 2 -> class 1
the class label 1 will be assigned to the sample.
1.11.5.1.1. Usage
The following example shows how to fit the majority rule classifier:
1.11.5.2. Weighted Average Probabilities (Soft Voting)
In contrast to majority voting (hard voting), soft voting returns the class label as argmax of the sum of predicted probabilities.
Specific weights can be assigned to each classifier via the weights
parameter. When weights are provided, the predicted class probabilities for each classifier are collected, multiplied by the classifier weight, and averaged. The final class label is then derived from the class label with the highest average probability.
To illustrate this with a simple example, let’s assume we have 3 classifiers and a 3-class classification problems where we assign equal weights to all classifiers: w1=1, w2=1, w3=1.
The weighted average probabilities for a sample would then be calculated as follows:
Here, the predicted class label is 2, since it has the highest average probability.
The following example illustrates how the decision regions may change when a soft VotingClassifier is used based on an linear Support Vector Machine, a Decision Tree, and a K-nearest neighbor classifier:
1.11.5.3. Using the VotingClassifier with GridSearch
The VotingClassifier can also be used together with GridSearch in order to tune the hyperparameters of the individual estimators:
1.11.5.3.1. Usage
In order to predict the class labels based on the predicted class-probabilities (scikit-learn estimators in the VotingClassifier must support predict_proba
method):
Optionally, weights can be provided for the individual classifiers:
- 1.11. 集成方法
- 集成方法
- 集成方法
- android集成phonegap方法
- 集成吊顶安装方法
- CocoaPOD集成方法
- 集成VVDocumenter的方法
- Ensemble methods(集成方法)
- VS2005集成VSS2005的方法
- VS2005集成VSS2005的方法
- VS2005集成VSS2005的方法
- vs2005集成vss2005的方法
- Turbine与Jbuilder集成方法
- Spring集成Struts的方法
- VS2005集成VSS2005的方法
- struts2 集成webservice 的方法
- java集成的构造方法
- Spring集成Struts的方法
- SpringSecurity源码解读(二) successHandler
- 后缀数组模版O(N*2个log)求Sa[i]+字符串查找
- 在Ubuntu/Linux环境下使用MySQL:安装MySQL
- React笔记
- unity profiler深入学习4:gpu usage profiler
- 1.11. 集成方法
- java程序实现开机自启动功能
- Spring Boot关于代码结构两点建议
- InputStreamReader和OutputStreamWriter转换流
- STM32F429 DMA串口数据发送
- 帝国cms评分、投票
- 高性能MySQL之Count统计查询
- 浅谈group by和order by的用法和区别
- JS定时模拟器。可模拟定时任务