XGBoost原理及在Python中使用XGBoost
来源:互联网 发布:手机挡字幕软件 编辑:程序博客网 时间:2024/04/19 21:20
原理见:http://www.myexception.cn/operating-system/2084839.html
译文转自:http://blog.csdn.net/zc02051126/article/details/46771793
在Python中使用XGBoost
下面将介绍XGBoost的Python模块,内容如下:
* 编译及导入Python模块
* 数据接口
* 参数设置
* 训练模型l
* 提前终止程序
* 预测
A walk through python example for UCI Mushroom dataset is provided.
安装
首先安装XGBoost的C++版本,然后进入源文件的根目录下的 wrappers
文件夹执行如下脚本安装Python模块
<code class="language-shell hljs cmake has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">python setup.py <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">install</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
安装完成后按照如下方式导入XGBoost的Python模块
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> xgboost <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">as</span> xgb</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
=
数据接口
XGBoost可以加载libsvm格式的文本数据,加载的数据格式可以为Numpy的二维数组和XGBoost的二进制的缓存文件。加载的数据存储在对象DMatrix
中。
- 加载libsvm格式的数据和二进制的缓存文件时可以使用如下方式
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">dtrain = xgb.DMatrix(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'train.svm.txt'</span>)dtest = xgb.DMatrix(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'test.svm.buffer'</span>)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>
- 加载numpy的数组到
DMatrix
对象时,可以用如下方式
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">data = np.random.rand(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># 5 entities, each contains 10 features</span>label = np.random.randint(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>, size=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># binary target</span>dtrain = xgb.DMatrix( data, label=label)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>
- 将
scipy.sparse
格式的数据转化为DMatrix
格式时,可以使用如下方式
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">csr = scipy.sparse.csr_matrix( (dat, (row,col)) )dtrain = xgb.DMatrix( csr )</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>
- 将
DMatrix
格式的数据保存成XGBoost的二进制格式,在下次加载时可以提高加载速度,使用方式如下
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">dtrain = xgb.DMatrix(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'train.svm.txt'</span>)dtrain.save_binary(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"train.buffer"</span>)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>
- 可以用如下方式处理
DMatrix
中的缺失值:
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">dtrain = xgb.DMatrix( data, label=label, missing = -<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">999.0</span>)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
- 当需要给样本设置权重时,可以用如下方式
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">w = np.random.rand(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>)dtrain = xgb.DMatrix( data, label=label, missing = -<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">999.0</span>, weight=w)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>
参数设置
XGBoost使用key-value格式保存参数. Eg
* Booster(基本学习器)参数
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">param = {<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'bst:max_depth'</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'bst:eta'</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'silent'</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'objective'</span>:<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'binary:logistic'</span> }param[<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'nthread'</span>] = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>plst = param.items()plst += [(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'eval_metric'</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'auc'</span>)] <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># Multiple evals can be handled in this way</span>plst += [(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'eval_metric'</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'ams@0'</span>)]</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul>
- 还可以定义验证数据集,验证算法的性能
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">evallist = [(dtest,<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'eval'</span>), (dtrain,<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'train'</span>)]</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
=
训练模型
有了参数列表和数据就可以训练模型了
* 训练
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">num_round = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>bst = xgb.train( plst, dtrain, num_round, evallist )</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>
- 保存模型
在训练完成之后可以将模型保存下来,也可以查看模型内部的结构
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">bst.save_model(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'0001.model'</span>)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
- Dump Model and Feature Map
You can dump the model to txt and review the meaning of model
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># dump model</span>bst.dump_model(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'dump.raw.txt'</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># dump model with feature map</span>bst.dump_model(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'dump.raw.txt'</span>,<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'featmap.txt'</span>)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>
- 加载模型
通过如下方式可以加载模型
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">bst = xgb.Booster({<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'nthread'</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>}) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#init model</span>bst.load_model(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"model.bin"</span>) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># load data</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>
=
提前终止程序
如果有评价数据,可以提前终止程序,这样可以找到最优的迭代次数。如果要提前终止程序必须至少有一个评价数据在参数evals
中。 If there’s more than one, it will use the last.
train(..., evals=evals, early_stopping_rounds=10)
The model will train until the validation score stops improving. Validation error needs to decrease at least every early_stopping_rounds
to continue training.
If early stopping occurs, the model will have two additional fields: bst.best_score
and bst.best_iteration
. Note that train()
will return a model from the last iteration, not the best one.
This works with both metrics to minimize (RMSE, log loss, etc.) and to maximize (MAP, NDCG, AUC).
=
Prediction
After you training/loading a model and preparing the data, you can start to do prediction.
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">data = np.random.rand(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># 7 entities, each contains 10 features</span>dtest = xgb.DMatrix( data, missing = -<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">999.0</span> )ypred = bst.predict( xgmat )</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>
If early stopping is enabled during training, you can predict with the best iteration.
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">ypred = bst.predict(xgmat,ntree_limit=bst.best_iteration)</code>
- XGBoost原理及在Python中使用XGBoost
- XGBoost:在Python中使用XGBoost
- XGBoost:在Python中使用XGBoost
- XGBoost:在Python中使用XGBoost
- 在Python中使用XGBoost
- 在Python中使用XGBoost
- Xgboost原理及使用
- xgboost原理及应用
- xgboost原理及应用
- xgboost原理及应用
- xgboost原理及应用
- xgboost原理及应用
- python 中 xgboost 安装
- 在R中使用XGBoost算法
- 在R中使用XGBoost算法
- xgboost原理
- xgboost原理
- xgboost原理
- 环境变量的配置
- hrbust 1453 AAAAHH! Overbooked!【水题】
- dp心得
- 对 Jenkins+ANT+Jmeter 接口测试的实践
- C++ 命名空间std 和 头文件
- XGBoost原理及在Python中使用XGBoost
- 散列表
- IMX6 ILI2302M(USB)触摸屏移植
- C++ 命名空间介绍和头文件介绍
- Page-encoding specified in XML prolog (UTF-8) is different from that specified in page directive (ut
- GridView中均分中碰到的几个问题
- 如何加快AS的运行速度
- VR Android播放器实现(一)
- QDP核心计算使用方法