scikit-learn:6. Strategies to scale computationally: bigger data
来源:互联网 发布:csgo帧数优化补丁 编辑:程序博客网 时间:2024/04/29 03:44
参考:http://scikit-learn.org/stable/modules/scaling_strategies.html
对于examples、features(或者两者)数量很大的情况,挑战传统的方法要解决两个问题:内存和效率。办法是Out-of-core (or “external memory”) learning。有三种方法可以实现out-of-core,分别是:
1、Streaming instances(流体化实例):
简单说就是,instances是一个一个来的。具体实现不在scikit-learn文档范围。
2、Extracting features:
简单说就是利用different feature extraction methods(翻译之后的文章:http://blog.csdn.net/mmc2015/article/details/46992105)实现大数据提取有用数据,简化内存、提高效率。不细讲。
3、Incremental learning:
all estimators implementing the partial_fit API are candidates。
the ability to learn incrementally from a mini-batch of instances (sometimes called “online learning”) is key to out-of-core learning as it guarantees that at any given time there will be only a small amount of instances in the main memory。
所有实现 partial_fit API 的estimators都可以实现增量学习,包括:
- Classification
- sklearn.naive_bayes.MultinomialNB
- sklearn.naive_bayes.BernoulliNB
- sklearn.linear_model.Perceptron
- sklearn.linear_model.SGDClassifier
- sklearn.linear_model.PassiveAggressiveClassifier
- Regression
- sklearn.linear_model.SGDRegressor
- sklearn.linear_model.PassiveAggressiveRegressor
- Clustering
- sklearn.cluster.MiniBatchKMeans
- Decomposition / feature Extraction
- sklearn.decomposition.MiniBatchDictionaryLearning
- sklearn.decomposition.IncrementalPCA
- sklearn.cluster.MiniBatchKMeans
注意:对于分类问题,由于incremental learner可能不知道所有的classes有哪些,所以第一次调用partial_fit时,最好人工设定参数 classes= ,指明所有类别。
4、Examples:
a example of Out-of-core classification of text documents. 通过例子可以更好理解上面的内容。
- scikit-learn:6. Strategies to scale computationally: bigger data
- scikit-learn Preprocessing data
- Scikit-learn Cookbook (二) --- Classifying Data with scikit-learn
- how to install scikit-learn
- Learn How To Realize Positive Risk Response Strategies (Opportunities)
- How To Handle Negative Risk-Learn Negative Risk Response Strategies
- Scikit-learn:数据预处理Preprocessing data
- From scikit-learn to Spark ML
- AirbnB uses R to scale data science
- scikit-learn
- scikit-learn
- scikit-learn
- scikit-learn
- scikit-learn
- scikit-learn
- [Scikit-Learn]
- scikit-learn
- scikit-learn
- SVN 常用命令
- 白皮书是什么意思?
- shell版俄罗斯方块一:方向键获取
- .net单元测试框架xUnit.net
- Linux Out-Of-Memory(OOM) Killer分析
- scikit-learn:6. Strategies to scale computationally: bigger data
- 普元启动时连接超时
- 设计模式学习笔记十一:观察者模式
- hdu 1875 畅通工程再续
- iOS本地相册 摄像头
- Dynamics CRM2013/2015 插件注册工具登录后无法显示assembly列表问题的解决办法
- 数学_LightOJ_1008
- 自动取款机 软件安装及操作故障总结
- Jenjins 一