数据预处理系列：（一）从外部源获取样本数据

来源：互联网发布：网络群发软件编辑：程序博客网时间：2024/06/11 12:35

博主简介：风雪夜归子（英文名: Allen），机器学习算法攻城狮，喜爱钻研Machine Learning的黑科技，对Deep Learning和Artificial Intelligence充满兴趣，经常关注kaggle数据挖掘竞赛平台，对数据、Machine Learning和Artificial Intelligence有兴趣的各位童鞋可以一起探讨哦，个人CSDN博客： http://blog.csdn.net/u013719780?viewmode=contents

从外部源获取样本数据

如果条件允许，学本系列内容时尽量用你熟悉的数据集；方便起见，我们用scikit-learn的内置数据库。这些内置数据库可用于测试不同的建模技术，如回归和分类。而且这些内置数据库都是非常著名的数据库。这对不同领域的学术论文的作者们来说是很用的，他们可以用这些内置数据库将他们的模型与其他模型进行比较。

推荐使用IPython来运行文中的指令。大内存很重要，这样可以让普通的命令正常运行。如果用IPython Notebook就更好了。如果你用Notebook，记得用%matplotlib inline指令，这样图象就会出现在Notebook里面，而不是一个新窗口里。

Getting ready

scikit-learn的内置数据库在datasets模块里。用如下命令导入：

In [1]:

from sklearn import datasetsimport numpy as np

在IPython里面运行datasets.*?就会看到datasets模块的指令列表。

How to do it…

datasets模块主要有两种数据类型。较小的测试数据集在sklearn包里面，可以通过datasets.load_*?查看。较大的数据集可以根据需要下载。后者默认情况下不在sklearn包里面；但是，有时这些大数据集可以更好的测试模型和算法，因为比较复杂足以模拟现实情形。

默认在sklearn包里面的数据集可以通过datasets.load_*?查看。另外一些数据集需要通过datasets.fetch_*?下载，这些数据集更大，没有被自动安装。经常用于测试那些解决实际问题的算法。

首先，加载boston数据集看看：

In [3]:

boston = datasets.load_boston()print(boston.DESCR)

Boston House Prices datasetNotes------Data Set Characteristics:      :Number of Instances: 506     :Number of Attributes: 13 numeric/categorical predictive        :Median Value (attribute 14) is usually the target    :Attribute Information (in order):        - CRIM     per capita crime rate by town        - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.        - INDUS    proportion of non-retail business acres per town        - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)        - NOX      nitric oxides concentration (parts per 10 million)        - RM       average number of rooms per dwelling        - AGE      proportion of owner-occupied units built prior to 1940        - DIS      weighted distances to five Boston employment centres        - RAD      index of accessibility to radial highways        - TAX      full-value property-tax rate per $10,000        - PTRATIO  pupil-teacher ratio by town        - B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town        - LSTAT    % lower status of the population        - MEDV     Median value of owner-occupied homes in $1000's    :Missing Attribute Values: None    :Creator: Harrison, D. and Rubinfeld, D.L.This is a copy of UCI ML housing dataset.http://archive.ics.uci.edu/ml/datasets/HousingThis dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonicprices and the demand for clean air', J. Environ. Economics & Management,vol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics...', Wiley, 1980.   N.B. Various transformations are used in the table onpages 244-261 of the latter.The Boston house-price data has been used in many machine learning papers that address regressionproblems.        **References**   - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.   - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.   - many more! (see http://archive.ics.uci.edu/ml/datasets/Housing)

DESCR将列出数据集的一些概况。下面我们来下载一个数据集：

In [4]:

housing = datasets.fetch_california_housing()print(housing.DESCR)

downloading Cal. housing from http://lib.stat.cmu.edu/modules.php?op=modload&name=Downloads&file=index&req=getit&lid=83 to C:\Users\tj2\scikit_learn_dataCalifornia housing dataset.The original database is available from StatLib    http://lib.stat.cmu.edu/The data contains 20,640 observations on 9 variables.This dataset contains the average house value as target variableand the following input variables (features): average income,housing average age, average rooms, average bedrooms, population,average occupation, latitude, and longitude in that order.References----------Pace, R. Kelley and Ronald Barry, Sparse Spatial Autoregressions,Statistics and Probability Letters, 33 (1997) 291-297.