Scikit-learn:模型评估Model evaluation
来源:互联网 发布:js时间戳获取月份 编辑:程序博客网 时间:2024/05/17 01:45
http://blog.csdn.net/pipisorry/article/details/52250760
模型评估Model evaluation: quantifying the quality of predictions
3 different approaches to evaluate the quality of predictions of a model:
- Estimator score method: Estimators have a score method providing a default evaluation criterion for the problem they are designed to solve. This is not discussed on this page, but in each estimator’s documentation.
- Scoring parameter: Model-evaluation tools using cross-validation (such as cross_validation.cross_val_scoreand grid_search.GridSearchCV) rely on an internal scoring strategy. This is discussed in the section The scoring parameter: defining model evaluation rules.
- Metric functions: The metrics module implements functions assessing prediction error for specific purposes. These metrics are detailed in sections on Classification metrics, Multilabel ranking metrics, Regression metrics and Clustering metrics.
Finally, Dummy estimators are useful to get a baseline value of those metrics for random predictions.
For “pairwise” metrics, between samples and not estimators or predictions, see the Pairwise metrics, Affinities and Kernels section.
sklearn一般评估方法
The scoring
parameter: defining model evaluation rules
Scoring Function Comment
metrics.accuracy_score
‘average_precision’metrics.average_precision_score
‘f1’metrics.f1_score
for binary targets‘f1_micro’metrics.f1_score
micro-averaged‘f1_macro’metrics.f1_score
macro-averaged‘f1_weighted’metrics.f1_score
weighted average‘f1_samples’metrics.f1_score
by multilabel sample‘neg_log_loss’metrics.log_loss
requires predict_proba
support‘precision’ etc.metrics.precision_score
suffixes apply as with ‘f1’‘recall’ etc.metrics.recall_score
suffixes apply as with ‘f1’‘roc_auc’metrics.roc_auc_score
Clustering ‘adjusted_rand_score’metrics.adjusted_rand_score
Regression ‘neg_mean_absolute_error’metrics.mean_absolute_error
‘neg_mean_squared_error’metrics.mean_squared_error
‘neg_median_absolute_error’metrics.median_absolute_error
‘r2’metrics.r2_score
使用如cross_val_score(clf, X, y, scoring='neg_log_loss') [Common cases: predefined values¶]
自定义scoring函数
>>> import numpy as np>>> def my_custom_loss_func(ground_truth, predictions):... diff = np.abs(ground_truth - predictions).max()... return np.log(1 + diff)...>>> # loss_func will negate the return value of my_custom_loss_func,>>> # which will be np.log(2), 0.693, given the values for ground_truth>>> # and predictions defined below.>>> loss = make_scorer(my_custom_loss_func, greater_is_better=False)>>> score = make_scorer(my_custom_loss_func, greater_is_better=True)>>> ground_truth = [[1, 1]]>>> predictions = [0, 1]>>> from sklearn.dummy import DummyClassifier>>> clf = DummyClassifier(strategy='most_frequent', random_state=0)>>> clf = clf.fit(ground_truth, predictions)>>> loss(clf,ground_truth, predictions) -0.69...>>> score(clf,ground_truth, predictions) 0.69...交叉验证自定义scoring函数
def rocAucScorer(*args): ''' 自定义ROC-AUC评价指标rocAucScorer(clf, x_test, y_true) :param y_true: y_test真值 :param x_test: x测试集 ''' from sklearn import metrics # y值比对函数 fun = lambda yt, ys: metrics.roc_auc_score([1.0 if _ > 0.0 else 0.0 for _ in yt], np.select([ys < 0.0, ys > 1.0, True], [0.0, 1.0, ys])) return metrics.make_scorer(fun, greater_is_better=True)(*args)[Defining your scoring strategy from metric functions¶]
或者You can generate even more flexible model scorers by constructing your ownscoring object from scratch, without using the make_scorer
factory.For a callable to be a scorer, it needs to meet the protocol specified bythe following two rules:
- It can be called with parameters
(estimator, X, y)
, whereestimator
is the model that should be evaluated,X
is validation data, andy
isthe ground truth target forX
(in the supervised case) orNone
(in theunsupervised case). - It returns a floating point number that quantifies the
estimator
prediction quality onX
, with reference toy
.Again, by convention higher numbers are better, so if your scorerreturns loss, that value should be negated.
def rocAucScorer(clf, x_test, y_true): ''' 自定义ROC-AUC评价指标 :param y_true: y_test真值 :param x_test: x测试集 ''' from sklearn import metrics ys = clf.predict(x_test) score = metrics.roc_auc_score([1.0 if _ > 0.0 else 0.0 for _ in y_true], np.select([ys < 0.0, ys > 1.0, True], [0.0, 1.0, ys])) return score
[Implementing your own scoring object¶]
皮皮blog
分类模型评估
分类模型评估方法
Some of these are restricted to the binary classification case:
matthews_corrcoef
(y_true, y_pred[, ...])Compute the Matthews correlation coefficient (MCC) for binary classesprecision_recall_curve
(y_true, probas_pred)Compute precision-recall pairs for different probability thresholdsroc_curve
(y_true, y_score[, pos_label, ...])Compute Receiver operating characteristic (ROC)Others also work in the multiclass case:
cohen_kappa_score
(y1, y2[, labels, weights])Cohen’s kappa: a statistic that measures inter-annotator agreement.confusion_matrix
(y_true, y_pred[, labels, ...])Compute confusion matrix to evaluate the accuracy of a classificationhinge_loss
(y_true, pred_decision[, labels, ...])Average hinge loss (non-regularized)Some also work in the multilabel case:
accuracy_score
(y_true, y_pred[, normalize, ...])Accuracy classification score.classification_report
(y_true, y_pred[, ...])Build a text report showing the main classification metricsf1_score
(y_true, y_pred[, labels, ...])Compute the F1 score, also known as balanced F-score or F-measurefbeta_score
(y_true, y_pred, beta[, labels, ...])Compute the F-beta scorehamming_loss
(y_true, y_pred[, labels, ...])Compute the average Hamming loss.jaccard_similarity_score
(y_true, y_pred[, ...])Jaccard similarity coefficient scorelog_loss
(y_true, y_pred[, eps, normalize, ...])Log loss, aka logistic loss or cross-entropy loss.precision_recall_fscore_support
(y_true, y_pred)Compute precision, recall, F-measure and support for each classprecision_score
(y_true, y_pred[, labels, ...])Compute the precisionrecall_score
(y_true, y_pred[, labels, ...])Compute the recallzero_one_loss
(y_true, y_pred[, normalize, ...])Zero-one classification loss.And some work with binary and multilabel (but not multiclass) problems:
average_precision_score
(y_true, y_score[, ...])Compute average precision (AP) from prediction scoresroc_auc_score
(y_true, y_score[, average, ...])Compute Area Under the Curve (AUC) from prediction scores[Classification metrics]分类准确率(Classification accuracy)
# calculate accuracyfrom sklearn import metricsprint metrics.accuracy_score(y_test, y_pred_class)0.692708333333
分类准确率分数
是指所有分类正确的百分比。
空准确率(null accuracy)
是指当模型总是预测比例较高的类别,那么其正确的比例是多少。
# examine the class distribution of the testing set (using a Pandas Series method)y_test.value_counts()
# calculate the percentage of onesy_test.mean()
# calculate the percentage of zeros1 - y_test.mean()
# calculate null accuracy(for binary classification problems coded as 0/1)max(y_test.mean(), 1-y_test.mean())
我们看到空准确率是68%,而分类准确率是69%,这说明该分类准确率并不是很好的模型度量方法,分类准确率的一个缺点是其不能表现任何有关测试数据的潜在分布。
# calculate null accuracy (for multi-class classification problems)y_test.value_counts().head(1) / len(y_test)
比较真实和预测的类别响应值:
# print the first 25 true and predicted responsesprint "True:", y_test.values[0:25]print "Pred:", y_pred_class[0:25]
从上面真实值和预测值的比较中可以看出,当正确的类别是0时,预测的类别基本都是0;当正确的类别是1时,预测的类别大都不是1。换句话说,该训练的模型大都在比例较高的那项类别的预测中预测正确,而在另外一中类别的预测中预测失败,而我们没法从分类准确率这项指标中发现这个问题。
分类准确率这一衡量分类器的标准比较容易理解,但是它不能告诉你响应值的潜在分布,并且它也不能告诉你分类器犯错的类型。接下来介绍的混淆矩阵可以识别这个问题。
基于混淆矩阵的评估度量
混淆矩阵
混淆矩阵赋予一个分类器性能表现更全面的认识,同时它通过计算各种分类度量,指导你进行模型选择。
# IMPORTANT: first argument is true values, second argument is predicted valuesprint metrics.confusion_matrix(y_test, y_pred_class)[[118 12] [ 47 15]]
- 真阳性(True Positive,TP):指被分类器正确分类的正例数据
- 真阴性(True Negative,TN):指被分类器正确分类的负例数据
- 假阳性(False Positive,FP):被错误地标记为正例数据的负例数据
- 假阴性(False Negative,FN):被错误地标记为负例数据的正例数据
准确率、识别率(Classification Accuracy):分类器正确分类的比例
print (TP+TN) / float(TP+TN+FN+FP)print metrics.accuracy_score(y_test, y_pred_class)
错误率、误分类率(Classification Error):分类器误分类的比例
print (FP+FN) / float(TP+TN+FN+FP)print 1-metrics.accuracy_score(y_test, y_pred_class)
考虑类不平衡问题,其中感兴趣的主类是稀少的。即数据集的分布反映负类显著地占多数,而正类占少数。故面对这种问题,需要其他的度量,评估分类器正确地识别正例数据的情况和正确地识别负例数据的情况。
召回率(Recall),也称为真正例识别率、灵敏性(Sensitivity)、查全率:正确识别的正例数据在实际正例数据中的百分比
print TP / float(TP+FN)print metrics.recall_score(y_test, y_pred_class)
特效性(Specificity),也称为真负例率:正确识别的负例数据在实际负例数据中的百分比
print TN / float(TN+FP)
假阳率(False Positive Rate):实际值是负例数据,预测错误的百分比
print FP / float(TN+FP)specificity = TN / float(TN+FP)print 1 - specificity
精度(Precision):看做精确性的度量,即标记为正类的数据实际为正例的百分比
print TP / float(TP+FP)precision = metrics.precision_score(y_test, y_pred_class)print precision
F度量(又称为F1分数或F分数),是使用精度和召回率的方法组合到一个度量上
print (2*precision*recall) / (precision+recall)print metrics.f1_score(y_test, y_pred_class)
sklearn评估报告
from sklearn import metrics# summarize the fit of the modelprint(metrics.classification_report(expected, predicted))计算的结果中有这些指标
precision recall f1-score support
0 0.96 0.77 0.86 4950
1 0.18 0.62 0.28 392
avg / total 0.90 0.76 0.81 5342
Note: 如0.18就是正例1对应的precision。
Precision-Recall曲线
sklearn.metrics.precision_recall_curve(y_true, probas_pred, pos_label=None, sample_weight=None)
参数
y_true : array, shape = [n_samples]
True targets of binary classification in range {-1, 1} or {0, 1}. 也就是二分类的binary。
probas_pred : array, shape = [n_samples]
Estimated probabilities or decision function. 必须是连续的continuous.
def plotPR(yt, ys, title=None): ''' 绘制precision-recall曲线 :param yt: y真值 :param ys: y预测值, recall, ''' import seaborn from sklearn import metrics from matplotlib import pyplot as plt precision, recall, thresholds = metrics.precision_recall_curve(yt, ys) plt.plot(precision, recall, 'darkorange', lw=1, label='x=precision') plt.plot(recall, precision, 'blue', lw=1, label='x=recall') plt.legend(loc='best') plt.plot([0, 1], [0, 1], color='navy', linestyle='--') plt.title('Precision-Recall curve for %s' % title) plt.ylabel('Recall') plt.xlabel('Precision') plt.show() plt.savefig(os.path.join(CWD, 'middlewares/pr-' + title + '.png'))[sklearn.metrics.precision_recall_curve]
ROC曲线和AUC
[机器学习模型的评价指标和方法 :ROC曲线和AUC]
sklearn.metrics.roc_curve(y_true, y_score, pos_label=None, sample_weight=None, drop_intermediate=True)
参数
y_true : array, shape = [n_samples]
True targets of binary classification in range {-1, 1} or {0, 1}. 也就是二分类的binary。
y_score : array, shape = [n_samples]
Target scores, can either be probability estimates of the positiveclass, confidence values, or non-thresholded measure of decisions(as returned by “decision_function” on some classifiers).应该也必须是连续的continuous.
绘制实现
def plotRUC(yt, ys, title=None): ''' 绘制ROC-AUC曲线 :param yt: y真值 :param ys: y预测值 ''' from sklearn import metrics from matplotlib import pyplot as plt f_pos, t_pos, thresh = metrics.roc_curve(yt, ys) auc_area = metrics.auc(f_pos, t_pos) print('auc_area: {}'.format(auc_area)) plt.plot(f_pos, t_pos, 'darkorange', lw=2, label='AUC = %.2f' % auc_area) plt.legend(loc='lower right') plt.plot([0, 1], [0, 1], color='navy', linestyle='--') plt.title('ROC-AUC curve for %s' % title) plt.ylabel('True Pos Rate') plt.xlabel('False Pos Rate') plt.show() plt.savefig(os.path.join(CWD, 'middlewares/roc-' + title + '.png'))[
roc_curve
(y_true, y_score[, pos_label, ...])][roc_auc_score
(y_true, y_score[, average, ...])]
使用什么度量取决于具体的业务要求
- 垃圾邮件过滤器:优先优化精度或者特效性,因为该应用对假阳性(非垃圾邮件被放进垃圾邮件箱)的要求高于对假阴性(垃圾邮件被放进正常的收件箱)的要求
- 欺诈交易检测器:优先优化灵敏度,因为该应用对假阴性(欺诈行为未被检测)的要求高于假阳性(正常交易被认为是欺诈)的要求
皮皮blog
from: http://blog.csdn.net/pipisorry/article/details/52250760
ref: [机器学习模型的评价指标和方法 ]
[scikit-learn User Guide]
[Model selection and evaluation]
[3.3. Model evaluation: quantifying the quality of predictions]*
[3.5. Validation curves: plotting scores to evaluate models]*
- Scikit-learn:模型评估Model evaluation
- 转:Scikit-learn:模型评估Model evaluation
- Scikit-learn:模型评估Model evaluation 之绘图
- scikit-learn:3. Model selection and evaluation
- scikit-learn:3. Model selection and evaluation
- scikit-learn:3.3. Model evaluation: quantifying the quality of predictions
- scikit-learn : 线性回归模型性能评估
- Scikit-learn:模型选择Model selection
- 模型评估与选择(Model Evaluation and Selection)
- scikit-learn评估结果
- 【Scikit-Learn 中文文档】模型评估: 量化预测的质量
- 【Scikit-Learn 中文文档】模型评估: 量化预测的质量
- 【Scikit-Learn 中文文档】模型评估: 量化预测的质量
- 【Scikit-Learn 中文文档】模型评估: 量化预测的质量
- 【Scikit-Learn 中文文档】模型评估: 量化预测的质量
- 【Scikit-Learn 中文文档】模型评估: 量化预测的质量
- 【Scikit-Learn 中文文档】模型评估: 量化预测的质量
- 【Scikit-Learn 中文文档】模型评估: 量化预测的质量
- C#操作excel
- 2016最流行的Java工具
- 《第一行代码》复习四 自定义控件
- java设计模式之四
- AndroidManifest.xml uses-feature 详解
- Scikit-learn:模型评估Model evaluation
- Bundler的简单安装
- 树链剖分 ZJOI2008 树的统计 Count
- 用Flask-mail发送邮件失败解决方案
- kaggle之电影文本情感分类
- excel动态的生成图片
- TTS务必使用expdp(数据泵)来做,不要使用exp来做
- (开发笔记)转载:MySQL 5.6 for Windows 解压缩版配置安装
- 计算机网络笔记--拥塞控制和流量控制区别分析