[Hadoop培训笔记]03-MapReduce集群的安装与部署
来源:互联网 发布:手机midi制作软件 编辑:程序博客网 时间:2024/05/16 14:46
注:开源力量Hadoop Development网络培训,链接:http://new.osforce.cn/course/52 个人笔记,不具参考性。
说明:
1)实验采用的是hadoop1.x,hadoop2.x的代码做一个分析
2)学习hadoop的基础知识;java基础;Linux操作基础
MapRedcue集群环境搭建
MapReduce WordCount运行
Hadoop Eclipse插件
bin目录下执行:./start-mapred.sh,启动jobtracker和tasktracker
问题:启动后使用jps命令,看不到上述tracker在运行。
原因:通过logs目录里的hadoop-michaelchen-tasktracker-mars.clustertech.com.log,查看后发现这么一条:Can not start task tracker because java.lang.IllegalArgumentException: Does not contain a valid host:port authority: local
解决方法:没有配置 mapred-site.xml,配置如下。再次启动,使用jps可以看到jobtracker和tasktracker
<property> <name>mapred.job.tracker</name> <value>localhost:9101</value> </property>
bin目录下创建 vim a.txt,里面写一些单词作为输入文件。
1 hello world 2 hello java 3 java c++
然后将a.txt放入一个创建的input目录里,让hadoop example中的wordcount程序执行,从input目录里获取数据,结果放到output目录里
./hadoop fs -mkdir /input./hadoop fs -put a.txt /input./hadoop jar ../hadoop-examples-1.2.1.jar wordcount /input /output
执行的过程比较有趣,所以也贴到这里
13/12/04 12:23:07 INFO input.FileInputFormat: Total input paths to process : 113/12/04 12:23:07 INFO util.NativeCodeLoader: Loaded the native-hadoop library13/12/04 12:23:07 WARN snappy.LoadSnappy: Snappy native library not loaded13/12/04 12:23:08 INFO mapred.JobClient: Running job: job_201312041206_000113/12/04 12:23:09 INFO mapred.JobClient: map 0% reduce 0%13/12/04 12:23:22 INFO mapred.JobClient: map 100% reduce 0%13/12/04 12:23:34 INFO mapred.JobClient: map 100% reduce 100%13/12/04 12:23:38 INFO mapred.JobClient: Job complete: job_201312041206_000113/12/04 12:23:38 INFO mapred.JobClient: Counters: 2913/12/04 12:23:38 INFO mapred.JobClient: Job Counters13/12/04 12:23:38 INFO mapred.JobClient: Launched reduce tasks=113/12/04 12:23:38 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=1419913/12/04 12:23:38 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=013/12/04 12:23:38 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=013/12/04 12:23:38 INFO mapred.JobClient: Launched map tasks=113/12/04 12:23:38 INFO mapred.JobClient: Data-local map tasks=113/12/04 12:23:38 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=1216813/12/04 12:23:38 INFO mapred.JobClient: File Output Format Counters13/12/04 12:23:38 INFO mapred.JobClient: Bytes Written=2913/12/04 12:23:38 INFO mapred.JobClient: FileSystemCounters13/12/04 12:23:38 INFO mapred.JobClient: FILE_BYTES_READ=5113/12/04 12:23:38 INFO mapred.JobClient: HDFS_BYTES_READ=13413/12/04 12:23:38 INFO mapred.JobClient: FILE_BYTES_WRITTEN=11731313/12/04 12:23:38 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=2913/12/04 12:23:38 INFO mapred.JobClient: File Input Format Counters13/12/04 12:23:38 INFO mapred.JobClient: Bytes Read=3213/12/04 12:23:38 INFO mapred.JobClient: Map-Reduce Framework13/12/04 12:23:38 INFO mapred.JobClient: Map output materialized bytes=5113/12/04 12:23:38 INFO mapred.JobClient: Map input records=313/12/04 12:23:38 INFO mapred.JobClient: Reduce shuffle bytes=5113/12/04 12:23:38 INFO mapred.JobClient: Spilled Records=813/12/04 12:23:38 INFO mapred.JobClient: Map output bytes=5613/12/04 12:23:38 INFO mapred.JobClient: Total committed heap usage (bytes)=18114150413/12/04 12:23:38 INFO mapred.JobClient: CPU time spent (ms)=540013/12/04 12:23:38 INFO mapred.JobClient: Combine input records=613/12/04 12:23:38 INFO mapred.JobClient: SPLIT_RAW_BYTES=10213/12/04 12:23:38 INFO mapred.JobClient: Reduce input records=413/12/04 12:23:38 INFO mapred.JobClient: Reduce input groups=413/12/04 12:23:38 INFO mapred.JobClient: Combine output records=413/12/04 12:23:38 INFO mapred.JobClient: Physical memory (bytes) snapshot=17591500813/12/04 12:23:38 INFO mapred.JobClient: Reduce output records=413/12/04 12:23:38 INFO mapred.JobClient: Virtual memory (bytes) snapshot=78720614413/12/04 12:23:38 INFO mapred.JobClient: Map output records=6
看看创建了什么文件,使用命令 ./hadoop fs -lsr /
drwxr-xr-x - michaelchen supergroup 0 2013-12-04 12:20 /input-rw-r--r-- 1 michaelchen supergroup 32 2013-12-04 12:20 /input/a.txtdrwxr-xr-x - michaelchen supergroup 0 2013-12-04 12:23 /output-rw-r--r-- 1 michaelchen supergroup 0 2013-12-04 12:23 /output/_SUCCESSdrwxr-xr-x - michaelchen supergroup 0 2013-12-04 12:23 /output/_logsdrwxr-xr-x - michaelchen supergroup 0 2013-12-04 12:23 /output/_logs/history-rw-r--r-- 1 michaelchen supergroup 13815 2013-12-04 12:23 /output/_logs/history/job_201312041206_0001_1386130987951_michaelchen_word+count-rw-r--r-- 1 michaelchen supergroup 49533 2013-12-04 12:23 /output/_logs/history/job_201312041206_0001_conf.xml-rw-r--r-- 1 michaelchen supergroup 29 2013-12-04 12:23 /output/part-r-00000drwxr-xr-x - michaelchen supergroup 0 2013-12-04 10:26 /systemdrwxr-xr-x - michaelchen supergroup 0 2013-12-04 12:17 /tmpdrwxr-xr-x - michaelchen supergroup 0 2013-12-04 12:17 /tmp/hadoop-michaelchendrwxr-xr-x - michaelchen supergroup 0 2013-12-04 12:23 /tmp/hadoop-michaelchen/mapreddrwxr-xr-x - michaelchen supergroup 0 2013-12-04 12:23 /tmp/hadoop-michaelchen/mapred/stagingdrwxr-xr-x - michaelchen supergroup 0 2013-12-04 12:23 /tmp/hadoop-michaelchen/mapred/staging/michaelchendrwx------ - michaelchen supergroup 0 2013-12-04 12:23 /tmp/hadoop-michaelchen/mapred/staging/michaelchen/.stagingdrwx------ - michaelchen supergroup 0 2013-12-04 12:23 /tmp/hadoop-michaelchen/mapred/system-rw------- 1 michaelchen supergroup 4 2013-12-04 12:17 /tmp/hadoop-michaelchen/mapred/system/jobtracker.infodrwxr-xr-x - michaelchen supergroup 0 2013-12-04 10:33 /userdrwxr-xr-x - michaelchen supergroup 0 2013-12-04 10:33 /user/michaelchendrwxr-xr-x - michaelchen supergroup 0 2013-12-04 10:33 /user/michaelchen/archiveDirdrwxr-xr-x - michaelchen supergroup 0 2013-12-04 10:33 /user/michaelchen/archiveDir/pack.har-rw-r--r-- 1 michaelchen supergroup 0 2013-12-04 10:33 /user/michaelchen/archiveDir/pack.har/_SUCCESS-rw-r--r-- 5 michaelchen supergroup 72 2013-12-04 10:33 /user/michaelchen/archiveDir/pack.har/_index-rw-r--r-- 5 michaelchen supergroup 22 2013-12-04 10:33 /user/michaelchen/archiveDir/pack.har/_masterindex-rw-r--r-- 1 michaelchen supergroup 15147 2013-12-04 10:33 /user/michaelchen/archiveDir/pack.har/part-0drwxr-xr-x - michaelchen supergroup 0 2013-12-04 10:31 /xwchen-rw-r--r-- 1 michaelchen supergroup 15147 2013-12-04 10:31 /xwchen/hadoop
分析后的结果放在这里,使用命令查看:./hadoop fs -cat /output/part-r-00000
c++ 1hello 2java 2world 1
Hadoop Eclipse插件
1)hadoop自0.20.x版本后不再提供现成的hadoop-eclipse插件,而是给出了源码自行编译
2)编译采用的是ant+ivy
3)URL:http://wiki.apache.org/hadoop/EclipsePlugin
编译的整个过程很是容易出问题,建议参考此链接:http://www.srccodes.com/p/article/30/build-hadoop-eclipse-plugin-jar-from-source-code-and-install-that-plugin-in-eclipse-ide
链接中注意版本号可能会有更改。
不过我编译的hadoop plugin的插件还是在eclipse中无法使用,最后用的网上下载的。。。
无法使用是出现在eclipse菜单里,show-others里编辑mapreduce路径,点new,弹不出窗口,泪奔啊~~
WordCount里用的相关Jar包依赖如下:
输入参数:hdfs://192.168.56.101:9100/input /output
注意output之前不能存在,否则报错。
其他参考视频。在1.2.1中编译成功,运行成功。
试题中的一些笔记:
- hadoop对于小文件的处理方式:Archive、CombinedInputFormat、SequenceFile、MapFile
- Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks.
- Borg(来自Google),YARN(来自Apache,属于Hadoop下面的一个分支,开源),Mesos(来自Twitter,开源),Torca(来自腾讯搜搜),Corona(来自Facebook,开源)一类系统被称为资源统一管理系统或者资源统一调度系统,它们是大数据时代的必然产物。
- SecondaryNamenode的作用是合并fsimage和editlogs
- Apache Tez由Hortonworks开发
- 关系型数据库与HDFS之间相互数据迁移的工具是sqoop
- hadoop2.x采用maven构建源代码
- [Hadoop培训笔记]03-MapReduce集群的安装与部署
- [Hadoop培训笔记]02-HDFS集群的安装与部署
- hadoop学习3-MapReduce的集群安装与部署
- Ubuntu与Centos的Hadoop安装以及编译运行MapReduce,Hadoop集群安装配置教程
- Hadoop集群部署笔记
- hadoop集群安装部署
- Hadoop集群安装部署
- hadoop集群安装部署
- 为hadoop集群安装mapreduce
- 安装部署Hadoop集群的方法
- Ganglia监控Hadoop集群的安装部署
- Ganglia监控Hadoop集群的安装部署
- Ganglia监控Hadoop集群的安装部署
- Ganglia监控Hadoop集群的安装部署
- Ganglia监控Hadoop集群的安装部署
- Ganglia监控Hadoop集群的安装部署
- hadoop 三个节点集群的安装部署
- hadoop集群的部署
- 依赖注入
- java.lang.ClassNotFoundException: [Ljava.lang.String解决办法
- 关于一些开源软件到了国内就成了闭源
- 可拖动、可关闭、透明标题的DIV弹出窗口,完美兼容各个浏览器
- 怎么把.cs文件导入VS c#项目中
- [Hadoop培训笔记]03-MapReduce集群的安装与部署
- 理解集合类
- Android手机在开发调试时logcat不显示输出信息的解决办法
- How to include <android/bitmap.h> and other *.h NDK libraries to Titanium Android module?
- java Date Calendar 日期函数
- 在VS环境下以ADO方式操作Oracle数据库
- 城市三级联动
- Servlet中的Filter过滤器的介绍和使用 编写过滤器
- 解决TMS Diagram Studio箭头颜色不能变化的问题