在MapReduce的Map和Reduce过程中使用自定义数据类型

来源：互联网发布：ubuntu自带中文输入法编辑：程序博客网时间：2024/05/17 03:02

手机流量数据文件如下所示，现在需要写一个MR程序计算每个手机的上行流量总数，下行流量总数及总流量。

1363157985066   13726230503 00-FD-07-A4-72-B8:CMCC  120.196.100.82  i02.c.aliimg.com        24  27  2481    24681   2001363157995052   13826544101 5C-0E-8B-C7-F1-E0:CMCC  120.197.40.4            4   0   264 0   2001363157991076   13926435656 20-10-7A-28-CC-0A:CMCC  120.196.100.99          2   4   132 1512    2001363154400022   13926251106 5C-0E-8B-8B-B1-50:CMCC  120.197.40.4            4   0   240 0   2001363157993044   18211575961 94-71-AC-CD-E6-18:CMCC-EASY 120.196.100.99  iface.qiyi.com  视频网站    15  12  1527    2106    2001363157995074   84138413    5C-0E-8B-8C-E8-20:7DaysInn  120.197.40.4    122.72.52.12        20  16  4116    1432    2001363157993055   13560439658 C4-17-FE-BA-DE-D9:CMCC  120.196.100.99          18  15  1116    954 2001363157995033   15920133257 5C-0E-8B-C7-BA-20:CMCC  120.197.40.4    sug.so.360.cn   信息安全    20  20  3156    2936    2001363157983019   13719199419 68-A1-B7-03-07-B1:CMCC-EASY 120.196.100.82          4   0   240 0   2001363157984041   13660577991 5C-0E-8B-92-5C-20:CMCC-EASY 120.197.40.4    s19.cnzz.com    站点统计    24  9   6960    690 2001363157973098   15013685858 5C-0E-8B-C7-F7-90:CMCC  120.197.40.4    rank.ie.sogou.com   搜索引擎    28  27  3659    3538    2001363157986029   15989002119 E8-99-C4-4E-93-E0:CMCC-EASY 120.196.100.99  www.umeng.com   站点统计    3   3   1938    180 2001363157992093   13560439658 C4-17-FE-BA-DE-D9:CMCC  120.196.100.99          15  9   918 4938    2001363157986041   13480253104 5C-0E-8B-C7-FC-80:CMCC-EASY 120.197.40.4            3   3   180 180 2001363157984040   13602846565 5C-0E-8B-8B-B6-00:CMCC  120.197.40.4    2052.flash2-http.qq.com 综合门户    15  12  1938    2910    2001363157995093   13922314466 00-FD-07-A2-EC-BA:CMCC  120.196.100.82  img.qfc.cn      12  12  3008    3720    2001363157982040   13502468823 5C-0A-5B-6A-0B-D4:CMCC-EASY 120.196.100.99  y0.ifengimg.com 综合门户    57  102 7335    110349  2001363157986072   18320173382 84-25-DB-4F-10-1A:CMCC-EASY 120.196.100.99  input.shouji.sogou.com  搜索引擎    21  18  9531    2412    2001363157990043   13925057413 00-1F-64-E1-E6-9A:CMCC  120.196.100.55  t3.baidu.com    搜索引擎    69  63  11058   48243   2001363157988072   13760778710 00-FD-07-A4-7B-08:CMCC  120.196.100.82          2   2   120 120 2001363157985066   13726238888 00-FD-07-A4-72-B8:CMCC  120.196.100.82  i02.c.aliimg.com        24  27  2481    24681   2001363157993055   13560436666 C4-17-FE-BA-DE-D9:CMCC  120.196.100.99          18  15  1116    954 200

思路

首先要将Map输入中的手机号，上行流量，下行流量数据抽取出来
Map的输出为<手机号,bean>。其中bean是自己封装的包含上行流量信息和下行流量信息的对象
Reduce在获得<手机号,list>后进行累积，然后输出结果即可。

代码

FlowSum.java

package tech.mrbcy.bigdata.mr.flowsum;import java.io.IOException;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.LongWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.mapreduce.Reducer;public class FlowSum {    static class FlowSumMapper extends Mapper<LongWritable,Text,Text,FlowBean>{        @Override        protected void map(LongWritable key, Text value, Context context)                throws IOException, InterruptedException {            String line = value.toString();            String[] arrs = line.split("\t");            String phoneNum = arrs[1];            Long upFlow = Long.parseLong(arrs[8]);            Long downFlow = Long.parseLong(arrs[9]);            context.write(new Text(phoneNum), new FlowBean(upFlow, downFlow));        }    }    static class FlowSumReducer extends Reducer<Text, FlowBean, Text, Text>{        @Override        protected void reduce(Text key, Iterable<FlowBean> values,Context context)                throws IOException, InterruptedException {            long sumUpFlow = 0;            long sumDownFlow = 0;            for(FlowBean bean : values){                sumDownFlow += bean.getDownFlow();                sumUpFlow += bean.getUpFlow();            }            String outValue = String.format("%d    %d    %d", sumUpFlow,sumDownFlow,(sumUpFlow+sumDownFlow));            context.write(key, new Text(outValue));        }    }    public static void main(String[] args) throws Exception {        Configuration conf = new Configuration();        Job job = Job.getInstance(conf,"Flow Sum");        job.setJarByClass(FlowSum.class);        job.setMapperClass(FlowSumMapper.class);        job.setReducerClass(FlowSumReducer.class);        job.setMapOutputKeyClass(Text.class);        job.setMapOutputValueClass(FlowBean.class);        job.setOutputKeyClass(Text.class);        job.setOutputValueClass(Text.class);        FileInputFormat.addInputPath(job, new Path(args[0]));        FileOutputFormat.setOutputPath(job, new Path(args[1]));        System.exit(job.waitForCompletion(true) ? 0 : 1);    }}

FlowBean.java

package tech.mrbcy.bigdata.mr.flowsum;import java.io.DataInput;import java.io.DataOutput;import java.io.IOException;import org.apache.hadoop.io.Writable;public class FlowBean implements Writable {    private long upFlow;    private long downFlow;    public FlowBean(){    }    public FlowBean(long upFlow, long downFlow) {        super();        this.upFlow = upFlow;        this.downFlow = downFlow;    }    public long getUpFlow() {        return upFlow;    }    public void setUpFlow(long upFlow) {        this.upFlow = upFlow;    }    public long getDownFlow() {        return downFlow;    }    public void setDownFlow(long downFlow) {        this.downFlow = downFlow;    }    @Override    /**     * 序列化方法     */    public void write(DataOutput out) throws IOException {        out.writeLong(upFlow);        out.writeLong(downFlow);    }    @Override    /**     * 反序列化方法     */    public void readFields(DataInput in) throws IOException {        upFlow = in.readLong();        downFlow = in.readLong();    }}

如果想在Reducer的输出结果中使用自定义的数据类型，重写FlowBean的toString()方法即可。

运行&总结

运行的过程不再赘述，最后的输出结果如下：

13480253104     180    180    36013502468823     7335    110349    11768413560436666     1116    954    207013560439658     2034    5892    792613602846565     1938    2910    484813660577991     6960    690    765013719199419     240    0    24013726230503     2481    24681    2716213726238888     2481    24681    2716213760778710     120    120    24013826544101     264    0    26413922314466     3008    3720    672813925057413     11058    48243    5930113926251106     240    0    24013926435656     132    1512    164415013685858     3659    3538    719715920133257     3156    2936    609215989002119     1938    180    211818211575961     1527    2106    363318320173382     9531    2412    1194384138413        4116    1432    5548

这个程序非常简单，但是它解决了如何在MapReduce程序中输入输出自定义的数据类型，因此具有较高的参考价值。

0 0