ubuntu下eclipse中hadoop中环境变量的设置

环境：ubuntu12.04（32位） eclipse-jee-luna-SR1-linux-gtk.tar.gz（32位） hadoop-2.2.0

1.安装hadoop

前一篇博客中有详细介绍 http://blog.csdn.net/nana_93/article/details/41912257

这里就不再详细说了

2.安装eclipse

下载软件：

解压后，将eclipse放在路径 /opt/中

tar -zxvf eclipse-jee-luna-SR1-linux-gtk.tar.gz

移动到/opt如下所示

给让当前用户分配opt/目录的权限

sudo chown hp:hp /opt/

进入eclipse的路径中，双击即可运行

3.在eclipse中配置hadoop环境

下载编译eclipse插件

或者直接下载已经编译好的插件：http://download.csdn.net/detail/zythy/6735167

配置hadoop插件

将下载的hadoop-eclipse-plugin-2.2.0.jar文件放到Eclipse的plugins目录下，重启Eclipse即可看到该插件已生效。

配置hadoop的安装路径windows->perspective

通过Open Perspective菜单打开Map Reduce视图，如下：

选中大象图标，右键点击New Hadoop Location编辑Hadoop配置信息：

填写正确的Map/Reduce和HDFS信息。（具体根据您的配置而定）

注意：MR Master和DFS Master配置必须和mapred-site.xml和core-site.xml等配置文件一致

打开Project Explorer,查看HDFS文件系统：

4.wordcount算法程序实现

1.新建Mapreduce项目

右键new一个WordCount的类

2.编写代码：

import java.io.IOException;import java.util.StringTokenizer;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.util.GenericOptionsParser;public class WordCount {public static class TokenizerMapper extends Mapper<Object,Text,Text,IntWritable>{private final static IntWritable one = new IntWritable(1);private Text word = new Text();public void map(Object key,Text value,Context context)throws IOException,InterruptedException{StringTokenizer itr = new StringTokenizer(value.toString());while(itr.hasMoreTokens()){word.set(itr.nextToken());context.write(word, one);}}public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable>{private IntWritable result = new IntWritable();public void reduce(Text key,Iterable<IntWritable> values,Context context) throws IOException,InterruptedException{int sum=0;for(IntWritable val:values){sum+=val.get();}result.set(sum);context.write(key,result);}}public static void main(String[] args) throws Exception {  Configuration conf = new Configuration();  String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();  if (otherArgs.length != 2) {  System.err.println("Usage: wordcount <in> <out>");  System.exit(2);  }  Job job = new Job(conf, "word count");job.setJarByClass(WordCount.class);   job.setMapperClass(TokenizerMapper.class);  job.setCombinerClass(IntSumReducer.class);job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class);    job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(otherArgs[0])); FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));System.exit(job.waitForCompletion(true) ? 0 : 1);  }  }}

3. 在外面新建一个文本文档word.txt，里面随便写点什么，如

java c++ python cjava c++ javascript helloworld hadoopmapreduce java hadoop hbase

然后在HDFS下新建文件夹 wordcount，将word.txt上传到wordcount下，如图所示。

hadoop fs -mkdir /tmp/wordcount

hadoop fs -copyFromLocal /home/hp/workspace/word.txt /tmp/wordcount/word.txt

    

   4. 运行项目

      选中WordCount，右击， Run AS--> Run Configuration.   在Java Application内new一个

是那个对话框内，Arguments那里写 hdfs://localhost:9000/tmp/wordcount/word.txt   hdfs://localhost:9000/tmp/wordcount/out

点击run

可能会出现类似如下的这种警告

log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).

解决办法

在src下面新建file名为log4j.properties内容如下:文件名就是这个，一个字也不能错

log4j.rootLogger=info,stdoutlog4j.appender.stdout=org.apache.log4j.ConsoleAppenderlog4j.appender.stdout.layout=org.apache.log4j.PatternLayout#dwr log configlog4j.logger.uk.ltd.getahead.dwr=# Pattern to output the caller's file name and line number.log4j.appender.stdout.layout.ConversionPattern=%5p [%t] (%c:%L) %d{yyyy-MM-dd HH:mm:ss,SSS} ---- %m%n#log4j.appender.R=org.apache.log4j.RollingFileAppender#log4j.appender.R.File=D:\\logs\\web_log.log#log4j.appender.R.MaxFileSize=100KB# Keep one backup file#log4j.appender.R.MaxBackupIndex=100log4j.appender.R.layout=org.apache.log4j.PatternLayoutlog4j.appender.R.layout.ConversionPattern=%p %d{yyyy-MM-dd HH\:mm\:ss,SSS} %n%t %c ---- %m%n#disable dwr log#log4j.logger.org.directwebremoting=ERROR

再次运行则不会出现警告了

运行结果