自己写wordcount程序在hadoop上运行

• 已经搭建好hadoop集群环境，并已成功在hadoop上运行wordcount程序，不过这里我将wordcount程序改了改，不做单词统计，做字符统计
• 下面是我的pom.xml

<properties>        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>        <hadoop.version>2.7.3</hadoop.version>    </properties>    <dependencies>        <dependency>            <groupId>org.apache.hadoop</groupId>            <artifactId>hadoop-common</artifactId>            <version>${hadoop.version}</version>        </dependency>        <dependency>            <groupId>org.apache.hadoop</groupId>            <artifactId>hadoop-hdfs</artifactId>            <version>${hadoop.version}</version>        </dependency>        <dependency>            <groupId>org.apache.hadoop</groupId>            <artifactId>hadoop-client</artifactId>            <version>${hadoop.version}</version>        </dependency>        <dependency>            <groupId>junit</groupId>            <artifactId>junit</artifactId>            <version>3.8.1</version>            <scope>test</scope>        </dependency>    </dependencies>

• 直接献上代码吧，参考hadoop2.7.3 在github上的代码修改部分而来

package com.lj.hadoop.hadoop.example;import java.io.IOException;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;public class CharCount {    public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>{        private final IntWritable one = new IntWritable(1);        private Text word = new Text();        @Override        public void map(Object key, Text value, Context context                ) throws IOException, InterruptedException {          String str = value.toString();          int len = str.length();          for (int i = 0; i < len; i++) {              char c = str.charAt(i);              if(!Character.isWhitespace(c)){                  word.set(String.valueOf(c));                  context.write(word, one);              }          }        }    }    public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable>{        private IntWritable result = new IntWritable();        @Override        protected void reduce(Text key, Iterable<IntWritable> values,                Context context) throws IOException, InterruptedException {            int sum = 0;            for (IntWritable one : values) {                sum += one.get();            }            result.set(sum);            context.write(key, result);        }    }    public static void main(String[] args) throws Exception{        Configuration cfg = new Configuration();        if(args.length < 2){            System.err.println("Usage: <in> <out>");            System.exit(2);        }        Job job = Job.getInstance(cfg, "myCharCount");        job.setJarByClass(CharCount.class);        job.setMapperClass(TokenizerMapper.class);        job.setCombinerClass(IntSumReducer.class);        job.setReducerClass(IntSumReducer.class);        job.setOutputKeyClass(Text.class);        job.setOutputValueClass(IntWritable.class);        FileInputFormat.addInputPath(job, new Path(args[0]));        FileOutputFormat.setOutputPath(job, new Path(args[1]));        System.exit(job.waitForCompletion(true) ? 0 : 1 );    }}

• 将程序打包，放到hadoop安装目录的share/hadoop/mapreduce/下，与hadoop官方的示例在同一目录
• 编辑一个文件并放到hdfs中

vim wordcount.txtHello AlfredHello WorldHello TomHello JackHello HadoopBye   hadoop

• 把文件放到hdfs根目录

hadoop fs -put wordcount.txt /

• 执行以下命令，运行程序

hadoop jar /usr/local/hadoop/hadoop/share/hadoop/mapreduce/hadoop-example-0.0.1.jar com.lj.hadoop.hadoop.example.CharCount /wordcount.txt /test/out-charcount-my

• 程序执行过程

  

• 执行成功，到hdfs查看结果