hadoop平台搭建详解

1.准备Linux环境

          

       1.1修改主机名

              vim/etc/sysconfig/network

             

              NETWORKING=yes

              HOSTNAME=wangyanqi    ###

 

       1.2修改IP

              两种方式：

              第一种：通过Linux图形界面进行修改（强烈推荐）

                     进入Linux图形界面 -> 右键点击右上方的两个小电脑 -> 点击Edit connections -> 选中当前网络System eth0 -> 点击edit按钮 -> 选择IPv4 -> method选择为manual -> 点击add按钮 -> 添加IP：192.168.44.88 子网掩码：255.255.255.0 网关：192.168.1.1 -> apply

      

              第二种：修改配置文件方式（屌丝程序猿专用）

                     vim/etc/sysconfig/network-scripts/ifcfg-eth0

                    

                     DEVICE="eth0"

                     BOOTPROTO="static"           ###

                     HWADDR="00:0C:29:3C:BF:E7"

                     IPV6INIT="yes"

                     NM_CONTROLLED="yes"

                     ONBOOT="yes"

                     TYPE="Ethernet"

                     UUID="ce22eeca-ecde-4536-8cc2-ef0dc36d4a8c"

                     IPADDR="192.168.1.44"       ###

                     NETMASK="255.255.255.0"      ###

                     GATEWAY="192.168.1.1"        ###

                    

       1.3修改主机名和IP的映射关系

              vim/etc/hosts

                    

              127.0.0.1   localhost localhost.localdomain localhost4localhost4.localdomain4

::1         localhostlocalhost.localdomain localhost6 localhost6.localdomain6

192.168.44.88   wangyanqi

      

       1.4关闭防火墙

              #查看防火墙状态

              serviceiptables status

              #关闭防火墙

              serviceiptables stop

              #查看防火墙开机启动状态

              chkconfigiptables --list

              #关闭防火墙开机启动

              chkconfigiptables off

      

       1.5重启Linux

              reboot

      

2.安装JDK

       2.1上传

      

       2.2解压jdk

              #创建文件夹

              mkdir usr/java

              #解压

              tar-zxvf jdk-7u71-linux-x64.gz -C /usr/java/ (进入文件所在目录解压)

             

       2.3将java添加到环境变量中

              vim/etc/profile

              #在文件最后添加

              exportJAVA_HOME=/usr/java/ jdk1.7.0_71

              exportPATH=$PATH:$JAVA_HOME/bin

      

              #刷新配置

              source/etc/profile

3.安装Hadoop

       3.1上传hadoop安装包

      

       3.2解压hadoop安装包

              mkdir/cloud

              #解压到/cloud/目录下

              tar-zxvf hadoop-2.5.2.tar.gz -C /root/cloud/

             

       3.3修改配置文件（5个）

              第一个：hadoop-env.sh

              #在27行修改

              exportJAVA_HOME=/usr/java/jdk1.7.0_71

             

              第二个：core-site.xml

              <configuration>

                     <!--指定HDFS老大（namenode）的通信地址 -->

                     <property>

                                   <name>fs.defaultFS</name>

                                   <value>hdfs://wangyanqi:9000</value>

                     </property>

                     <!--指定hadoop运行时产生文件的存储路径 -->

                     <property>

                                   <name>hadoop.tmp.dir</name>

                                   <value>/cloud/hadoop-2.5.2/tmp</value>

                     </property>

              </configuration>

             

              第三个：hdfs-site.xml

              <configuration>

                     <!--设置hdfs副本数量 -->

                     <property>

                                   <name>dfs.replication</name>

                                   <value>1</value>

                     </property>

              </configuration>

             

              第四个：mapred-site.xml.template 需要重命名： mv mapred-site.xml.templatemapred-site.xml

              <configuration>

                     <!--通知框架MR使用YARN -->

                     <property>

                                   <name>mapreduce.framework.name</name>

                                   <value>yarn</value>

                     </property>

              </configuration>

             

              第五个：yarn-site.xml

              <configuration>

                     <!--reducer取数据的方式是mapreduce_shuffle-->

                     <property>

                            <name>yarn.nodemanager.aux-services</name>

                            <value>mapreduce_shuffle</value>

                     </property>

              </configuration>

      

       3.4将hadoop添加到环境变量

              vim/etc/profile

             

                exportJAVA_HOME=/usr/java/jdk1.7.0_71

exportPATH=$PATH:$JAVA_HOME/bin

exportHADOOP_HOME=/root/cloud/hadoop-2.5.2

exportPATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH   

              source/etc/profile

       3.5格式化HDFS（namenode）第一次使用时要格式化（进入hadoop）

              hadoopnamenode –format

             

       3.6启动hadoop

              先启动HDFS

              sbin/start-dfs.sh

             

              再启动YARN

              sbin/start-yarn.sh

             

       3.7验证是否启动成功

              使用jps命令验证

              27408NameNode

              28218Jps

              27643SecondaryNameNode

              28066NodeManager

              27803ResourceManager

              27512DataNode

             

4.配置ssh免登陆

       生成ssh免登陆密钥

       cd~，进入到我的home目录

       cd.ssh/

 

       ssh-keygen-t rsa （四个回车）

       执行完这个命令后，会生成两个文件id_rsa（私钥）、id_rsa.pub（公钥）

       将公钥拷贝到要免登陆的机器上

       cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

       或

       ssh-copy-id -i wangyanqi