PIG安装使用

来源：互联网发布：合肥网络兼职信息编辑：程序博客网时间：2024/05/22 09:47

上一篇博客中记录了用hive执行mapreduce任务实现统计最大值以及总量，这里用另外一个强大的工具PIG实现同样的功能。

首先下载pig-0.10.1.tar.gz版本解压到hadoop/pig下面，配置好HADOOP_HOME以及PATH环境变量

#读取HDFS中的数据到变量中grunt> Line = load 'inputTest/test.log' using PigStorage(' ') as (day, bytes, tag, user);grunt> describe Line;Line: {day: bytearray,bytes: bytearray,tag: bytearray,user: bytearray}grunt> dump Line;(20121221,04567,user,s00001)(20121221,75531,user,s00003)(20121222,52369,user,s00002)(20121222,01297,user,s00001)(20121223,61223,user,s00002)(20121223,33121,user,s00003)#对变量进行group操作grunt> Groupd_Line = group Line by day;grunt> describe Groupd_Line;Groupd_Line: {group: bytearray,Line: {(day: bytearray,bytes: bytearray,tag: bytearray,user: bytearray)}}grunt> dump Groupd_Line;(20121221,{(20121221,04567,user,s00001),(20121221,75531,user,s00003)})(20121222,{(20121222,52369,user,s00002),(20121222,01297,user,s00001)})(20121223,{(20121223,61223,user,s00002),(20121223,33121,user,s00003)})#计算总量grunt> Sum_Groupd_Line = foreach Groupd_Line generate group, SUM(Line.bytes);grunt> describe Sum_Groupd_Line;Sum_Groupd_Line: {group: bytearray,double}grunt> dump Sum_Groupd_Line;(20121221,80098.0)(20121222,53666.0)(20121223,94344.0)#保存到HDFS中grunt> store Sum_Groupd_Line into 'sumOutput';#通过HDFS查看结果root:~/hadoop # hadoop fs -cat /user/root/sumOutput/part-r-0000020121221        80098.020121222        53666.020121223        94344.0#计算最大值也是类似操作grunt> Max_Groupd_Line = foreach Groupd_Line generate group, MAX(Line.bytes);grunt> dump Max_Groupd_Line;(20121221,75531.0)(20121222,52369.0)(20121223,61223.0)grunt> store Max_Groupd_Line into 'maxOutput';

对lzo压缩格式的文件一样的操作

grunt> Line = load 'inputTest/test.log.lzo' using PigStorage(' ') as (day, bytes, tag, user);grunt> Filt = FILTER Line by day == 20121221;grunt> dump Filt;(20121221,04567,user,s00001)(20121221,75531,user,s00003)

参考资料：pig的安装和使用