程序博客网 > nginx rewrite if

Hadoop 0.23.6安装实践1-单机开发版安装

来源：互联网 发布：nginx rewrite if 编辑：程序博客网 时间：2024/06/05 17:09

hadoop 0.23开始添加了yarn（MRv2）模块，文件结构也发生了较大的变化。以下就安装部署流程详细记录一下。

安装环境：

  1. 系统：Ubuntu 12.10
  2. hadoop：0.23.6
  3. jdk：sun 1.7.0_21

安装步骤：

一.安装JDK

安装 orcale jdk，并且配置环境以及设置成默认（略）

检查jdk是否正确安装和配置

在主目录下执行java -version

如果出现下面类似结果则ok

hadoop@ubuntu:~$ java -version
java version "1.7.0_21"
Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
Java HotSpot(TM) Server VM (build 23.21-b01, mixed mode)

二.安装hadoop

1.下载:

http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-0.23.6/hadoop-0.23.6.tar.gz

2.安装

tar -zxvf hadoop-0.23.6.tar.gz

mv hadoop /opt/

cd /opt

sudo ln -s /opt/hadoop-0.23.6 /opt/hadoop

三.安装ssh server

sudo apt-get install openssh-server

四.添加hadoop用户

为了方便hadoop的管理，最好添加一个单独的用户来管理hadoop，例如添加hadoop用户

执行以下命令

sudo adduser hadoop

然后会提示输入密码，设置好密码就可以了

这时候只要执行

su hadoop

输入密码后就可以切换到hadoop用户下

注：

为了使hadoop帐号下能够sudo，建议做以下添加

在：

%sudo ALL=(ALL:ALL) ALL 后添加
hadoop ALL=(ALL:ALL) ALL

五.配置本机ssh无密码登录

hadoop@ubuntu:~$ ssh-keygen-t rsa-P ""

hadoop@ubuntu:~$ cat~/.ssh/id_rsa.pub>>~/.ssh/authorized_keys

测试：

hadoop@ubuntu:~$ ssh localhost
Welcome to Ubuntu 12.10 (GNU/Linux 3.5.0-17-generic i686)

* Documentation: https://help.ubuntu.com/

340 packages can be updated.
105 updates are security updates.

Last login: Thu Apr 18 07:18:03 2013 from localhost

六.配置Hadoop

chown -R hadoop:hadoop /opt/hadoop

chown -R hadoop:hadoop /opt/hadoop-0.23.6

su hadoop

1. 配置jdk及hadoop环境变量

在~/.bashrc文件里追加

export JAVA_HOME=/usr/lib/jvm/java-7-sun
export JRE_HOME=${JAVA_HOME}/jre
export HADOOP_HOME=/opt/hadoop
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$HADOOP_HOME/bin:$PATH
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

2. Hadoop文件配置

hadoop@ubuntu:~$ cd /opt/hadoop/etc/hadoop/

hadoop@ubuntu:/opt/hadoop/etc/hadoop$ vi yarn-env.sh

追加以下配置

export HADOOP_FREFIX=/opt/hadoop
export HADOOP_COMMON_HOME=${HADOOP_FREFIX}
export HADOOP_HDFS_HOME=${HADOOP_FREFIX}
export PATH=$PATH:$HADOOP_FREFIX/bin
export PATH=$PATH:$HADOOP_FREFIX/sbin
export HADOOP_MAPRED_HOME=${HADOOP_FREFIX}
export YARN_HOME=${HADOOP_FREFIX}
export HADOOP_CONF_HOME=${HADOOP_FREFIX}/etc/hadoop
export YARN_CONF_DIR=${HADOOP_FREFIX}/etc/hadoop
vi core-site.xml

Xml代码 复制代码 收藏代码
  1. <configuration>
  2. <property>
  4. <name>fs.defaultFS</name>
  6. <value>hdfs://localhost:12200</value>
  8. </property>
  10. <property>
  12. <name>hadoop.tmp.dir</name>
  14. <value>/opt/hadoop/hadoop-root</value>
  16. </property>
  18. <property>
  20. <name>fs.arionfs.impl</name>
  22. <value>org.apache.hadoop.fs.pvfs2.Pvfs2FileSystem</value>
  24. <description>The FileSystem for arionfs.</description>
  26. </property>
  27. </configuration>
hadoop@ubuntu:/opt/hadoop/etc/hadoop$ vi hdfs-site.xml

Xml代码 复制代码 收藏代码
  1. <configuration>
  2. <property>
  4. <name>dfs.namenode.name.dir</name>
  6. <value>file:/opt/hadoop/data/dfs/name</value>
  8. <final>true</final>
  10. </property>
  12. <property>
  14. <name>dfs.namenode.data.dir</name>
  16. <value>file:/opt/hadoop/data/dfs/data</value>
  18. <final>true</final>
  20. </property>
  22. <property>
  24. <name>dfs.replication</name>
  26. <value>1</value>
  28. </property>
  30. <property>
  32. <name>dfs.permission</name>
  34. <value>false</value>
  36. </property>
  37. </configuration>
hadoop@ubuntu:/opt/hadoop/etc/hadoop$ vi mapred-site.xml

Xml代码 复制代码 收藏代码
  1. <configuration>
  2. <property>
  4. <name>mapreduce.framework.name</name>
  6. <value>yarn</value>
  8. </property>
  10. <property>
  12. <name>mapreduce.job.tracker</name>
  14. <value>hdfs://localhost:9001</value>
  16. <final>true</final>
  18. </property>
  20. <property>
  22. <name>mapreduce.map.memory.mb</name>
  24. <value>1536</value>
  26. </property>
  28. <property>
  30. <name>mapreduce.map.java.opts</name>
  32. <value>-Xmx1024M</value>
  34. </property>
  36. <property>
  38. <name>mapreduce.reduce.memory.mb</name>
  40. <value>3072</value>
  42. </property>
  44. <property>
  46. <name>mapreduce.reduce.java.opts</name>
  48. <value>-Xmx2560M</value>
  50. </property>
  52. <property>
  54. <name>mapreduce.task.io.sort.mb</name>
  56. <value>512</value>
  58. </property>
  59. <property>
  61. <name>mapreduce.task.io.sort.factor</name>
  63. <value>100</value>
  65. </property>
  67. <property>
  69. <name>mapreduce.reduce.shuffle.parallelcopies</name>
  71. <value>50</value>
  73. </property>
  75. <property>
  77. <name>mapreduce.system.dir</name>
  79. <value>file:/opt/hadoop/data/mapred/system</value>
  81. </property>
  83. <property>
  85. <name>mapreduce.local.dir</name>
  87. <value>file:/opt/hadoop/data/mapred/local</value>
  89. <final>true</final>
  91. </property>
  92. </configuration>
hadoop@ubuntu:/opt/hadoop/etc/hadoop$ vi yarn-site.xml

Xml代码 复制代码 收藏代码
  1. <configuration>
  3. <!-- Site specific YARN configuration properties -->
  4. <property>
  6. <name>yarn.nodemanager.aux-services</name>
  8. <value>mapreduce.shuffle</value>
  10. </property>
  12. <property>
  14. <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
  16. <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  18. </property>
  20. <property>
  22. <name>mapreduce.framework.name</name>
  24. <value>yarn</value>
  26. </property>
  28. <property>
  30. <name>user.name</name>
  32. <value>hadoop</value>
  34. </property>
  36. <property>
  38. <name>yarn.resourcemanager.address</name>
  40. <value>localhost:54311</value>
  42. </property>
  44. <property>
  46. <name>yarn.resourcemanager.scheduler.address</name>
  48. <value>localhost:54312</value>
  50. </property>
  51. <property>
  53. <name>yarn.resourcemanager.webapp.address</name>
  55. <value>localhost:54313</value>
  57. </property>
  59. <property>
  61. <name>yarn.resourcemanager.resource-tracker.address</name>
  63. <value>localhost:54314</value>
  65. </property>
  67. <property>
  69. <name>yarn.web-proxy.address</name>
  71. <value>localhost:54315</value>
  73. </property>
  75. <property>
  77. <name>mapred.job.tracker</name>
  79. <value>localhost</value>
  81. </property>
  82. </configuration>
到此所有配置完成,其中上述的localhost可以改成成本机的ip或者host，至于配置里面的属性待后续相关文章叙述。

七.启动并运行wordcount程序

1.设置JAVA_HOME

Java代码 复制代码 收藏代码
  1. hadoop@ubuntu:/opt/hadoop$ vi libexec/hadoop-config.sh
  2. 在
  3. if [[ -z $JAVA_HOME ]]; then
  4. # On OSX use java_home (or /Library for older versions)
  5. if [ "Darwin" == "$(uname -s)" ]; then
  6. if [ -x /usr/libexec/java_home ]; then
  7. export JAVA_HOME=($(/usr/libexec/java_home))
  8. else
  9. export JAVA_HOME=(/Library/Java/Home)
  10. fi
  11. fi
  13. # Bail if we did not detect it
  14. if [[ -z $JAVA_HOME ]]; then
  15. echo "Error: JAVA_HOME is not set and could not be found."1>&2
  16. exit 1
  17. fi
  18. fi
  20. 之前添加
  21. export JAVA_HOME=/usr/lib/jvm/java-7-sun
2. 格式化namenode

Java代码 复制代码 收藏代码
  1. hadoop@ubuntu:/opt/hadoop$ hadoop namenode -format
3.启动

Java代码 复制代码 收藏代码
  1. hadoop@ubuntu:/opt/hadoop/sbin$ ./start-dfs.sh
  2. Starting namenodes on [localhost]
  3. localhost: starting namenode, logging to /opt/hadoop-0.23.6/logs/hadoop-hadoop-namenode-ubuntu.out
  4. localhost: starting datanode, logging to /opt/hadoop-0.23.6/logs/hadoop-hadoop-datanode-ubuntu.out
  5. Starting secondary namenodes [0.0.0.0]
  6. 0.0.0.0: starting secondarynamenode, logging to /opt/hadoop-0.23.6/logs/hadoop-hadoop-secondarynamenode-ubuntu.out
  8. hadoop@ubuntu:/opt/hadoop/sbin$ ./start-yarn.sh
  9. starting yarn daemons
  10. starting resourcemanager, logging to /opt/hadoop-0.23.6/logs/yarn-hadoop-resourcemanager-ubuntu.out
  11. localhost: starting nodemanager, logging to /opt/hadoop-0.23.6/logs/yarn-hadoop-nodemanager-ubuntu.out
4.检查启动是否成功

Java代码 复制代码 收藏代码
  1. hadoop@ubuntu:/opt/hadoop/sbin$ jps
  2. 5036 DataNode
  3. 5246 SecondaryNameNode
  4. 5543 NodeManager
  5. 5369 ResourceManager
  6. 4852 NameNode
  7. 5816 Jps
5.试着运行wordcount

1)构造输入数据

生成一个字符文本文件

Java代码 复制代码 收藏代码
  1. hadoop@ubuntu:/opt/hadoop$ cat tmp/test.txt
  2. a c b a b d f f e b a c c d g i s a b c d e a b f g e i k m m n a b d g h i j a k j e

2)上传到hdfs

Java代码 复制代码 收藏代码
  1. hadoop@ubuntu:/opt/hadoop$ hadoop fs -mkdir /test
  2. hadoop@ubuntu:/opt/hadoop$ hadoop fs -copyFromLocal tmp/test.txt /test
  3. hadoop@ubuntu:/opt/hadoop$ hadoop fs -ls /test
  4. Found 1 items
  5. -rw-r--r-- 1 hadoop supergroup 86 2013-04-1807:47 /test/test.txt

3)执行程序

Shell代码 复制代码 收藏代码
  1. hadoop@ubuntu:/opt/hadoop$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-0.23.6.jar wordcount /test/test.txt /test/out #其中/test/out 为输出目录
  3. 13/04/1822:41:11 INFO input.FileInputFormat: Total input paths to process :1
  4. 13/04/1822:41:11 INFO util.NativeCodeLoader: Loaded the native-hadoop library
  5. 13/04/1822:41:11 WARN snappy.LoadSnappy: Snappy native library not loaded
  6. 13/04/1822:41:12 INFO mapreduce.JobSubmitter: number of splits:1
  7. 13/04/1822:41:12 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
  8. 13/04/1822:41:12 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
  9. 13/04/1822:41:12 WARN conf.Configuration: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
  10. 13/04/1822:41:12 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
  11. 13/04/1822:41:12 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
  12. 13/04/1822:41:12 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
  13. 13/04/1822:41:12 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
  14. 13/04/1822:41:12 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
  15. 13/04/1822:41:12 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
  16. 13/04/1822:41:12 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
  17. 13/04/1822:41:12 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
  18. 13/04/1822:41:13 INFO mapred.ResourceMgrDelegate: Submitted application application_1366295287642_0001 to ResourceManager at localhost/127.0.0.1:54311
  19. 13/04/1822:41:13 INFO mapreduce.Job: The url to track the job: http://localhost:54315/proxy/application_1366295287642_0001/
  20. 13/04/1822:41:13 INFO mapreduce.Job: Running job: job_1366295287642_0001
  21. 13/04/1822:41:21 INFO mapreduce.Job: Job job_1366295287642_0001 running in uber mode : false
  22. 13/04/1822:41:21 INFO mapreduce.Job: map0% reduce0%
  23. 13/04/1822:41:36 INFO mapreduce.Job: map100% reduce0%
  24. 13/04/1822:41:36 INFO mapreduce.Job: Task Id : attempt_1366295287642_0001_m_000000_0, Status : FAILED
  26. Killed by external signal
  28. 13/04/1822:41:37 INFO mapreduce.Job: map0% reduce0%
  29. 13/04/1822:42:11 INFO mapreduce.Job: map100% reduce0%
  30. 13/04/1822:42:26 INFO mapreduce.Job: map100% reduce100%
  31. 13/04/1822:42:26 INFO mapreduce.Job: Job job_1366295287642_0001 completed successfully
  32. 13/04/1822:42:27 INFO mapreduce.Job: Counters:45

4)查看结果

Shell代码 复制代码 收藏代码
  1. hadoop@ubuntu:/opt/hadoop$ hadoop fs -ls /test
  2. Found 2 items
  3. drwxr-xr-x - hadoop supergroup 0 2013-04-18 22:42 /test/out
  4. -rw-r--r-- 1 hadoop supergroup 86 2013-04-1807:47 /test/test.txt
  5. hadoop@ubuntu:/opt/hadoop$ hadoop fs -ls /test/out
  6. Found 2 items
  7. -rw-r--r-- 1 hadoop supergroup 0 2013-04-1822:42 /test/out/_SUCCESS
  8. -rw-r--r-- 1 hadoop supergroup 56 2013-04-1822:42 /test/out/part-r-00000
  9. hadoop@ubuntu:/opt/hadoop$ hadoop fs -cat /test/out/part-r-00000
  10. 13/04/1822:45:25 INFO util.NativeCodeLoader: Loaded the native-hadoop library
  11. a 7
  12. b 6
  13. c 4
  14. d 4
  15. e 4
  16. f 3
  17. g 3
  18. h 1
  19. i 3
  20. j 2
  21. k 2
  22. m 2
  23. n 1
  24. s 1
nginx rewrite if nginx rewrite if
原创粉丝点击
热门IT博客
什么电子狗软件最好用什么电子狗软件最好用
四种会话跟踪技术php四种会话跟踪技术php
妮维雅q10面霜知乎
windows常用网络命令
27世纪人工智能系统
数控加工编程代码讲解
微易达网络推广
网络计划的绘制规则
jsp mysql网上商城
windows常用api文档
内网映射软件
淘宝优惠券返利
java 代码单元测试
舆情软件破解版
be my love网络歌手
医院数据统计
c windows assembly
哪个windows系统好用
阿里云核验单下载
姚班 知乎
热门问题 老师的惩罚 人脸识别 我在镇武司摸鱼那些年 重生之率土为王 我在大康的咸鱼生活 盘龙之生命进化 天生仙种 凡人之先天五行 春回大明朝 姑娘不必设防，我是瞎子 攻城掠地荀攸怎么过 荀氏八龙 长沟落月作品 长沟落月 觉醒2之落月森林 月落倾世 月落旧暮里 月落 月出月落 誓不为妾长沟落月 我只是个女三长沟落月 嫁给权臣之后长沟落月 嫁给权臣之后长钩落月 月蚀 蚀月纯黑作品 月蚀的假面 超念觉醒 蚀月纯黑 零月蚀的假面 嫁给食人城主 蚀月桃 蜍 蜍怎么读 蜍蟾的图片 月诗 月满轩尼诗 爆笑老师 星诗月 九月九日忆山东兄弟诗配画 守得云开见月明的整首诗 月诗风吟面膜 古老月行李白诗朗读 毕竞西湖六月中这首诗 中秋月诗 玲珑月 白云诗诗诗 月 诗 诗月 关于月的诗 咏月的诗 诗 元旦 咏月诗 过年的诗 新年的诗 月的诗

程序博客网，程序员的互联网技术博客家园。csdn论坛精品 msdn技术资料都在这里