hadoop-2.2.0+hbase-0.96.0测试集群搭建

1、环境介绍：ubuntu13.04虚拟机3台：master、s1、s2；jdk1.7.0_40；hadoop-2.2.0；hbase-0.96.0；

2、分别配置好三台机器的JAVA_HOME，hosts，hostname配置分别为master、s1、s2。

3、解压hadoop包，配置$HADOOP_HOME/ect/hadoop/hadoop-env.sh

4,、配置core-site.xml

<configuration>        <property>                <name>fs.defaultFS</name>                <value>hdfs://master:9000</value>        </property>       <property>                <name>io.file.buffer.size</name>                <value>131072</value>       </property>       <property>                <name>hadoop.tmp.dir</name>                <value>/usr/local/opt/hadoop-2.2.0/temp</value>                <description>Abase for other temporary directories.</description>       </property>       <property>               <name>hadoop.proxyuser.hduser.hosts</name>               <value>*</value>       </property>       <property>               <name>hadoop.proxyuser.hduser.groups</name>               <value>*</value>       </property></configuration>

5、配置hdfs-site.xml

<configuration><property>               <name>dfs.namenode.secondary.http-address</name>               <value>master:9001</value>        </property>        <property>                <name>dfs.namenode.name.dir</name>                <value>/usr/local/opt/hadoop-2.2.0/dfs/name</value>        </property>        <property>                 <name>dfs.datanode.data.dir</name>                 <value>/usr/local/opt/hadoop-2.2.0/dfs/data</value>        </property>        <property>                  <name>dfs.replication</name>                  <value>3</value>        </property>        <property>                  <name>dfs.webhdfs.enabled</name>                  <value>true</value>         </property></configuration>

6、配置yarn-site.xml

<configuration><property>    <name>yarn.resourcemanager.resource-tracker.address</name>    <value>master:8031</value>    <description>host is the hostname of the resource manager and    port is the port on which the NodeManagers contact the Resource Manager.    </description>  </property>  <property>    <name>yarn.resourcemanager.scheduler.address</name>    <value>master:8030</value>    <description>host is the hostname of the resourcemanager and port is the port    on which the Applications in the cluster talk to the Resource Manager.    </description>  </property>  <property>    <name>yarn.resourcemanager.scheduler.class</name>    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>    <description>In case you do not want to use the default scheduler</description>  </property>  <property>    <name>yarn.resourcemanager.address</name>    <value>master:8032</value>    <description>the host is the hostname of the ResourceManager and the port is the port on    which the clients can talk to the Resource Manager. </description>  </property>  <property>    <name>yarn.nodemanager.local-dirs</name>    <value>${hadoop.tmp.dir}/nodemanager/local</value>    <description>the local directories used by the nodemanager</description>  </property>  <property>    <name>yarn.nodemanager.address</name>    <value>master:8034</value>    <description>the nodemanagers bind to this port</description>  </property>  <property>    <name>yarn.nodemanager.resource.memory-mb</name>    <value>10240</value>    <description>the amount of memory on the NodeManager in GB</description>  </property>  <property>    <name>yarn.nodemanager.remote-app-log-dir</name>    <value>${hadoop.tmp.dir}/nodemanager/remote</value>    <description>directory on hdfs where the application logs are moved to </description>  </property>   <property>    <name>yarn.nodemanager.log-dirs</name>    <value>${hadoop.tmp.dir}/nodemanager/logs</value>    <description>the directories used by Nodemanagers as log directories</description>  </property>  <property>    <name>yarn.nodemanager.aux-services</name>    <value>mapreduce.shuffle</value>    <description>shuffle service that needs to be set for Map Reduce to run </description>  </property></configuration>

7、配置yarn-env.sh里面的JAVA_HOME

8、配置slaves.写入s1、s2。

此时需求的工作差不多已经完成。以上所有步骤在master机器上进行。接下来配置3台机器之间的ssh免密码登录。

1、在master执行命令 ssh-keygen -t rsa -P ''，继续回车。之后在.ssh/下生成id_rsa.pub和id_rsa。然后将id_rsa.pub发送到s1的.ssh/下，并执行cat id_rsa.pub >>authorized_keys，设置authorized_keys权限为600。

2、从master直接ssh到s1，第一次要确认下，输入yes。此时就可以从master免密码登录s1。如果要互相免密码登录，就在不同的机器做以上操作，登录机器的公钥要被追加到被登录机器。

经过以上工作之后，scp发送配置好的hadoop到s1和s2.然后在master上启动hadoop。

1、首先格式化文件系统，./hdfs namenode -format。

2、启动 ./sbin/start-yarn.sh,./sbin/start-dfs.sh，成功之后会有namenode 和secondarynamenode的进程，在s1和s2会有datanode的进程。

接下来安装hbase-0.96.0

1、解压hbase-0.96.0-hadoop2-bin.tar.gz，配置$HBASE_HOME/conf/hbase-env.sh

export JAVA_HOME=/usr/local/opt/jdk1.7.0_40

export HBASE_MANAGES_ZK=true，这里使用hbase内部zookeeper，也可自行安装

2、配置hbase-site.xml

<configuration><property>    <name>hbase.rootdir</name>   <value>hdfs://master:9000/hbase</value>  </property>  <property>    <name>hbase.cluster.distributed</name>    <value>true</value>  </property>    <property>      <name>hbase.zookeeper.quorum</name>      <value>master,s1,s2</value>    </property><property>        <name>hbase.master</name>        <value>master:60000</value>        <description>The host and port that the HBase master runs at.</description></property></configuration>

3、配置regionservers，内容为s1、s2

4、替换hbase下的

hadoop-common-2.1.0-beta.jar
hadoop-hdfs-2.1.0-beta.jar
hadoop-auth-2.1.0-beta.jar

为hadoop下的

hadoop-common-2.2.0.jar

hadoop-hdfs-2.2.0.jar

hadoop-auth-2.2.0.jar

否则会启动报错，或者HMaster进程挂掉。

5、scp配置好的hbase到s1和s2

6、在master执行启动命令：./start-hbase.sh 成功后通过jps查看如下

可通过master:60010查看web界面