1.12. Multiclass and multilabel algorithms 多分类多标签算法

1.12. Multiclass and multilabel algorithms

Warning

 

All classifiers in scikit-learn do multiclass classification out-of-the-box. You don’t need to use thesklearn.multiclass module unless you want to experiment with different multiclass strategies.

The sklearn.multiclass module implements meta-estimators to solve multiclass and multilabel classification problems by decomposing such problems into binary classification problems.

• Multiclass classification means a classification task with more than two classes; e.g., classify a set of images of fruits which may be oranges, apples, or pears. Multiclass classification makes the assumption that each sample is assigned to one and only one label: a fruit can be either an apple or a pear but not both at the same time.

• 多类分类是指具有两个以上类的分类任务；例如，对一组水果可能是橙子、苹果或梨进行分类。多类分类的假设是，每个样本被分配到一个和唯一一个标签：一个水果可以是苹果或梨，但不是同时两个。
  

• Multilabel classification assigns to each sample a set of target labels. This can be thought as predicting properties of a data-point that are not mutually exclusive, such as topics that are relevant for a document. A text might be about any of religion, politics, finance or education at the same time or none of these.

• Multilabel classification分配给每个样品一套目标标签。这可以被认为是预测数据集的属性，这些数据集是不相互排斥的，例如与文档相关的主题。文本可能是关于任何宗教、政治、金融或教育的同时，或没有这些。
  

• Multioutput-multiclass classification and multi-task classification means that a single estimator has to handle several joint classification tasks. This is a generalization of the multi-label classification task, where the set of classification problem is restricted to binary classification, and of the multi-class classification task. The output format is a 2d numpy array or sparse matrix.

  The set of labels can be different for each output variable. For instance a sample could be assigned “pear” for an output variable that takes possible values in a finite set of species such as “pear”, “apple”, “orange” and “green” for a second output variable that takes possible values in a finite set of colors such as “green”, “red”, “orange”, “yellow”...

  This means that any classifiers handling multi-output multiclass or multi-task classification task supports the multi-label classification task as a special case. Multi-task classification is similar to the multi-output classification task with different model formulations. For more information, see the relevant estimator documentation.

• Multioutput-multiclass classification and multi-task classification意味着单一的评估器能够具有处理多个节点分类任务的能力。这是多标签分类任务的一个推广，将分类问题限定为二元分类和多类分类任务。输出格式是一个二维的NumPy数组或稀疏矩阵。每个输出变量的标签集可能不同。例如，样品可以分“梨”为输出变量的可能的值，以一组有限的物种如“梨”、“苹果”、“橙色”和“绿色”的第二个输出变量的可能的值，以一组有限的颜色，如“绿色”、“红色”、“橙色”、“黄色”…这意味着任何处理多输出多类或多任务分类任务的分类器都支持作为一种特殊情况的多标签分类任务。多任务分类类似于多输出分类任务，具有不同的模型表示。有关更多信息，请参见相关的估计文档。

All scikit-learn classifiers are capable of multiclass classification, but the meta-estimators offered by sklearn.multiclasspermit changing the way they handle more than two classes because this may have an effect on classifier performance (either in terms of generalization error or required computational resources).

Below is a summary of the classifiers supported by scikit-learn grouped by strategy; you don’t need the meta-estimators in this class if you’re using one of these unless you want custom multiclass behavior:

• Inherently multiclass: Naive Bayes, LDA and QDA, Decision Trees, Random Forests, Nearest Neighbors, setting multi_class='multinomial' in sklearn.linear_model.LogisticRegression.
• Support multilabel: Decision Trees, Random Forests, Nearest Neighbors, Ridge Regression.
• One-Vs-One: sklearn.svm.SVC.
• One-Vs-All: all linear models except sklearn.svm.SVC.

Some estimators also support multioutput-multiclass classification tasks Decision Trees, Random Forests, Nearest Neighbors.

Warning

 

At present, no metric in sklearn.metrics supports the multioutput-multiclass classification task.

1.12.1. Multilabel classification format  多分类数据标签label的转换

In multilabel learning, the joint set of binary classification tasks is expressed with label binary indicator array: each sample is one row of a 2d array of shape (n_samples, n_classes) with binary values: the one, i.e. the non zero elements, corresponds to the subset of labels. An array such as np.array([[1, 0, 0], [0, 1, 1], [0, 0, 0]]) represents label 0 in the first sample, labels 1 and 2 in the second sample, and no labels in the third sample.

Producing multilabel data as a list of sets of labels may be more intuitive. The MultiLabelBinarizer transformer can be used to convert between a collection of collections of labels and the indicator format.

>>> from sklearn.preprocessing import MultiLabelBinarizer>>> y = [[2, 3, 4], [2], [0, 1, 3], [0, 1, 2, 3, 4], [0, 1, 2]]>>> MultiLabelBinarizer().fit_transform(y)array([[0, 0, 1, 1, 1],       [0, 0, 1, 0, 0],       [1, 1, 0, 1, 0],       [1, 1, 1, 1, 1],       [1, 1, 1, 0, 0]])

1.12.2. One-Vs-The-Rest  一对多

This strategy, also known as one-vs-all, is implemented in OneVsRestClassifier. The strategy consists in fitting one classifier per class. For each classifier, the class is fitted against all the other classes. In addition to its computational efficiency (only n_classes classifiers are needed), one advantage of this approach is its interpretability. Since each class is represented by one and one classifier only, it is possible to gain knowledge about the class by inspecting its corresponding classifier. This is the most commonly used strategy and is a fair default choice.

1.12.2.1. Multiclass learning

Below is an example of multiclass learning using OvR:

>>> from sklearn import datasets>>> from sklearn.multiclass import OneVsRestClassifier>>> from sklearn.svm import LinearSVC>>> iris = datasets.load_iris()>>> X, y = iris.data, iris.target>>> OneVsRestClassifier(LinearSVC(random_state=0)).fit(X, y).predict(X)array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,       0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,       1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1,       1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 1, 2, 2, 2, 2,       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

1.12.2.2. Multilabel learning

OneVsRestClassifier also supports multilabel classification. To use this feature, feed the classifier an indicator matrix, in which cell [i, j] indicates the presence of label j in sample i.

Examples:

• Multilabel classification

1.12.3. One-Vs-One

OneVsOneClassifier constructs one classifier per pair of classes. At prediction time, the class which received the most votes is selected. In the event of a tie (among two classes with an equal number of votes), it selects the class with the highest aggregate classification confidence by summing over the pair-wise classification confidence levels computed by the underlying binary classifiers.

Since it requires to fit n_classes * (n_classes - 1) / 2 classifiers, this method is usually slower than one-vs-the-rest, due to its O(n_classes^2) complexity. However, this method may be advantageous for algorithms such as kernel algorithms which don’t scale well with n_samples. This is because each individual learning problem only involves a small subset of the data whereas, with one-vs-the-rest, the complete dataset is used n_classes times.

1.12.3.1. Multiclass learning

Below is an example of multiclass learning using OvO:

>>> from sklearn import datasets>>> from sklearn.multiclass import OneVsOneClassifier>>> from sklearn.svm import LinearSVC>>> iris = datasets.load_iris()>>> X, y = iris.data, iris.target>>> OneVsOneClassifier(LinearSVC(random_state=0)).fit(X, y).predict(X)array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,       0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,       1, 2, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1,       1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

References:

[1]“Pattern Recognition and Machine Learning. Springer”, Christopher M. Bishop, page 183, (First Edition)

1.12.4. Error-Correcting Output-Codes

Output-code based strategies are fairly different from one-vs-the-rest and one-vs-one. With these strategies, each class is represented in a euclidean space, where each dimension can only be 0 or 1. Another way to put it is that each class is represented by a binary code (an array of 0 and 1). The matrix which keeps track of the location/code of each class is called the code book. The code size is the dimensionality of the aforementioned space. Intuitively, each class should be represented by a code as unique as possible and a good code book should be designed to optimize classification accuracy. In this implementation, we simply use a randomly-generated code book as advocated in [3] although more elaborate methods may be added in the future.

At fitting time, one binary classifier per bit in the code book is fitted. At prediction time, the classifiers are used to project new points in the class space and the class closest to the points is chosen.

In OutputCodeClassifier, the code_size attribute allows the user to control the number of classifiers which will be used. It is a percentage of the total number of classes.

A number between 0 and 1 will require fewer classifiers than one-vs-the-rest. In theory, log2(n_classes) / n_classes is sufficient to represent each class unambiguously. However, in practice, it may not lead to good accuracy since log2(n_classes) is much smaller than n_classes.

A number greater than than 1 will require more classifiers than one-vs-the-rest. In this case, some classifiers will in theory correct for the mistakes made by other classifiers, hence the name “error-correcting”. In practice, however, this may not happen as classifier mistakes will typically be correlated. The error-correcting output codes have a similar effect to bagging.

1.12.4.1. Multiclass learning

Below is an example of multiclass learning using Output-Codes:

>>> from sklearn import datasets>>> from sklearn.multiclass import OutputCodeClassifier>>> from sklearn.svm import LinearSVC>>> iris = datasets.load_iris()>>> X, y = iris.data, iris.target>>> clf = OutputCodeClassifier(LinearSVC(random_state=0),...                            code_size=2, random_state=0)>>> clf.fit(X, y).predict(X)array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,       0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1,       1, 2, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 2, 2, 2, 1, 1, 1, 1, 1, 1,       1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,       2, 2, 2, 2, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 1, 1, 2, 2, 2,       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])