对阿里离线赛的数据用python进行简单的预处理

数据预处理

对阿里离线赛的数据用python进行预处理,
参考文章http://www.wyblog.cn/2016/12/16/%e5%a4%a9%e6%b1%a0%e4%bd%93%e9%aa%8c%e4%ba%8c-%e6%96%b0%e4%ba%ba%e7%a6%bb%e7%ba%bf%e8%b5%9b%e6%95%b0%e6%8d%ae%e5%8f%af%e8%a7%86%e5%8c%96%e5%88%86%e6%9e%90/

源数据为天池新人实战赛之[离线赛]的数据
tianchi_fresh_comp_train_user.csv
大概有1G那么大,处理为标准数据后后有1.3G,我是加了1个4G的内存条之后才能处理的

hadoop@master:/media/hadoop/娱乐/1离线赛/原始数据$ head -n 10 '/media/hadoop/娱 乐/1离线赛/原始数据/tianchi_fresh_comp_train_user (复件).csv' user_id,item_id,behavior_type,user_geohash,item_category,time10001082,285259775,1,97lk14c,4076,2014-12-08 1810001082,4368907,1,,5503,2014-12-12 1210001082,4368907,1,,5503,2014-12-12 1210001082,53616768,1,,9762,2014-12-02 1510001082,151466952,1,,5232,2014-12-12 1110001082,53616768,4,,9762,2014-12-02 1510001082,290088061,1,,5503,2014-12-12 1210001082,298397524,1,,10894,2014-12-12 1210001082,32104252,1,,6513,2014-12-12 12hadoop@master:/media/hadoop/娱乐/1离线赛/原始数据$ 

##导入包import pandas as pdimport numpy as np##读入数据user_df = pd.read_csv('/media/hadoop/娱乐/1离线赛/原始数据/tianchi_fresh_comp_train_user (复件).csv')##定义由time属性得到day和hour属性的函数def get_date(time):    date,hour = time.split(" ")    return datedef get_hour(time):    date,hour = time.split(" ")    return hour##用map()函数得到date和hour字段,利用pandas的to_datetime()date转换为datetime类型,user_df['date'] = user_df['time'].map(get_date)user_df['date'] = pd.to_datetime(user_df['date'])##并用astype()将hour转换为intuser_df['hour'] = user_df['time'].map(get_hour)user_df['hour'] = user_df['hour'].astype(int)##用pandas的get_dummies()对behavoir_type进行独热编码,生成#对behavior使用独热编码,方便后面的统计和使用#pandas的get_dummies方法可以将一个类似枚举的属性自动生成独热编码生成DataFramebehavior_type_dummies = pd.get_dummies(user_df['behavior_type'])##并为它的列命名behavior_type_dummies.columns = ['view','favor','cart','buy']##删除user_df中的无用字段,axis=1指定按列删除user_df.drop(['behavior_type','time','user_geohash'],axis=1,inplace=True)##用user_df和behavior_type_dummies二者join生成新的user_df就是想要的数据形式了user_df = user_df.join(behavior_type_dummies)##将生成的user_df存储到本地user_df.to_csv('/media/hadoop/娱乐/1离线赛/原始数据/user_df.csv')

可以看看生成的beahvior_type_dummies和user_df

behavior_type_dummies.head()   view  favor  cart  buy0     1      0     0    01     1      0     0    02     1      0     0    03     1      0     0    04     1      0     0    0user_df.head()    user_id    item_id  item_category       date  hour  view  favor  cart  buy0  10001082  285259775           4076 2014-12-08    18     1      0     0    01  10001082    4368907           5503 2014-12-12    12     1      0     0    02  10001082    4368907           5503 2014-12-12    12     1      0     0    03  10001082   53616768           9762 2014-12-02    15     1      0     0    04  10001082  151466952           5232 2014-12-12    11     1      0     0    0