Hadoop 0.20.205.0 + zookeeper + hbase安装历程

来源：互联网发布：php培训课程安排编辑：程序博客网时间：2024/05/17 22:47

一、规划

a) 集群名称：rock

b) rock-hdpnn 192.168.15.59 master

rock-hdp1 192.168.15.60 slaves

rock-hdp2 192.168.15.61 slaves

c) 网络配置

i. 修改主机名（root用户，其他机器做类似修改）

echo "rock-hdpnn" > /etc/hostname

ii. 增加主机名解析(root用户)

echo "192.168.15.59 rock-hdpnn" >> /etc/hosts

echo "192.168.15.60 rock-hdp1" >> /etc/hosts

echo "192.168.15.61 rock-hdp2" >> /etc/hosts

备注：配置完毕后需要重启生效（记得可以不用重启，忘记怎么搞了）

二、创建hadoop专用用户

用户/密码：hadoop/hadoop

命令：useradd -d /home/hadoop -s /bin/bash -m hadoop

passwd hadoop

三、ssh配置

a) 授信关系

rock-hdpnn --> rock-hdp1

rock-hdpnn --> rock-hdp2

b) 操作步骤

1. ssh-keygen -t rsa ----- 也可以用dsa

2. 确认密码文件存放位置，直接Enter

3. 输入口令，直接Enter，即口令为空

4. cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

5. 拷贝id_rsa.pub到所要授信的机器上，注意不要覆盖原先的文件

scp ~/.ssh/id_rsa.pub hadoop@192.168.15.60:~/.ssh/tmp (需要创建tmp文件夹)

6. 在所要授信的机器上将此id_rsa.pub内容追加到authorized_keys中

cat ~/.ssh/tmp/id_rsa.pub >> ~/.ssh/authorized_keys

7. 测试： ssh rock-hdp1 （第一次需要输入回车，最好都先手动测试一下）

8. 清理环境，删除tmp文件夹

四、安装配置JAVA环境

a) 去oracle官网下载JDK：

https://edelivery.oracle.com/otn-pub/java/jdk/7u25-b15/jdk-7u25-linux-x64.tar.gz

b) 解压至home目录

c) 配置，将一下环境变量写入~/.bashrc

export JAVA_HOME=/home/hadoop/jdk1.7.0_25

export CLASSPATH=.:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools.jar:${JAVA_HOME}/lib:${CLASSPATH}

export PATH=${JAVA_HOME}/bin:${PATH}

d) 重新登录，执行java -version检测是否安装成功

五、安装部署Hadoop

a) 去官网下载hadoop

http://archive.apache.org/dist/hadoop/core/hadoop-0.20.205.0/hadoop-0.20.205.0.tar.gz

b) 解压至home目录，并创建链接

ln -s hadoop-0.20.205.0 hadoop_home

c) 环境变量配置，将以下写入~/.bashrc

# config for hadoop

export HADOOP_HOME=/home/hadoop/hadoop_home

export PATH=${HADOOP_HOME}/bin:${PATH}

d) Hadoop配置

1. ${HADOOP_HOME}/conf/hadoop-env.sh

export JAVA_HOME=/home/hadoop/jdk1.7.0_25

export HADOOP_HOME=/home/hadoop/hadoop_home

export HADOOP_LOG_DIR=${HADOOP_HOME}/../log/hadoop

2. core-site.xml

<configuration>    <property>        <name>fs.default.name</name>        <value>hdfs://rock-hdpnn:49175</value>    </property>    <property>        <name>hadoop.tmp.dir</name>        <value>/home/${user.name}/hadoop_data</value>    </property></configuration>

3. hdfs-site.xml

<configuration>    <property>        <name>dfs.name.dir</name>        <value>/home/hadoop/disk0/name</value>    </property>    <property>        <name>dfs.data.dir</name>        <value>/home/hadoop/disk0/data</value>    </property>    <property>        <name>dfs.replication</name>        <value>1</value>    </property></configuration>

4. mapred-site.xml

<configuration>    <property>        <name>mapred.job.tracker</name>        <value>rock-hdpnn:49176</value>    </property>    <property>        <name>mapred.local.dir</name>        <value>/home/hadoop/disk0/mapred/local</value>    </property></configuration>

5. masters (暂时不配置secondary namenode)

echo "" > masters

6. salves

echo "rock-hdp1" > slaves

echo "rock-hdp2" >> slaves

e) 启动测试

必须先初始化HDFS：bin/hadoop namenode -format

启动/停止：start-all.sh/stop-all.sh

查看进程：利用jps查看各节点的进程

执行mapreduce：hadoop jar hadoop-examples-0.20.205.0.jar org.apache.hadoop.examples.WordCount input output

用浏览器查看：http://192.168.15.59:50070 (HDFS的namenode，默认端口为50070)

http://192.168.15.59:50030 (MapReduce的JobTracker，默认端口为50030)

六、安装部署Zookeeper

a) 下载:

http://mirrors.hust.edu.cn/apache/zookeeper/zookeeper-3.4.5/zookeeper-3.4.5.tar.gz

b) 解压，并创建链接：ln –s zookeeper-3.4.5 zookeeper_home

c) 环境变量配置，将以下写入.bashrc

export ZOOKEEPER_HOME=/home/hadoop/zookeeper_home

export PATH=${ZOOKEEPER_HOME}/bin:${PATH}

d) zookeeper配置

1. 拷贝一份zoo_sample.cfg为zoo.cfg

2. 修改zoo.cfg

dataDir=/home/hadoop/zookeeper_data

dataLogDir=/home/hadoop/log/zookeeper

clientPort=49178

server.0=rock-hdpnn:22888:23888

e) 启动zookeeper （编写启动脚本start-zk.sh）

#! /bin/bashexport ZOOKEEPER_HOME=/home/hadoop/zookeeper_homeexport ZOOKEEPER_BIN=$ZOOKEEPER_HOME/binexport ZOO_CONF_DIR=$ZOOKEEPER_HOME/confexport ZOO_LOG_DIR="~/log/zookeeper"export ZOO_DATA_DIR="~/zookeeper_data"zk_id_servers=`grep ^server. ${ZOO_CONF_DIR}/zoo.cfg | awk -F ":" '{print $1}'`for zk_id_server in $zk_id_serversdo        zk_id=`echo "${zk_id_server}" | awk -F "=" '{print $1}'| awk -F "." '{print $2}'`         zk_server=`echo "${zk_id_server}" | awk -F "=" '{print $2}'`         remote_cmd="export JAVA_HOME=${JAVA_HOME} && export ZOO_LOG_DIR=${ZOO_LOG_DIR} && echo ${zk_id} > ${ZOO_DATA_DIR}/myid && bash ${ZOOKEEPER_BIN}/zkServer.sh start"        ssh ${zk_server} "${remote_cmd}"done

stop-zk.sh

#! /bin/bashexport ZOOKEEPER_HOME=/home/hadoop/zookeeper_homeexport ZOOKEEPER_BIN=$ZOOKEEPER_HOME/binexport ZOO_CONF_DIR=$ZOOKEEPER_HOME/confexport ZOO_LOG_DIR="~/log/zookeeper"export ZOO_DATA_DIR="~/zookeeper_data"zk_id_servers=`grep ^server. ${ZOO_CONF_DIR}/zoo.cfg | awk -F ":" '{print $1}'`for zk_id_server in $zk_id_serversdo        zk_server=`echo "${zk_id_server}" | awk -F "=" '{print $2}'`         remote_cmd="export JAVA_HOME=${JAVA_HOME} && export ZOO_LOG_DIR=${ZOO_LOG_DIR} && bash ${ZOOKEEPER_BIN}/zkServer.sh stop && rm -f ${ZOO_DATA_DIR}/myid"        ssh ${zk_server} "${remote_cmd}"done

七、安装部署HBase

a) 下载：

http://archive.apache.org/dist/hbase/hbase-0.94.2/hbase-0.94.2.tar.gz

b) 解压，并创建链接：ln –s hbase-0.94.2 hbase_home

c) 环境变量配置

export HBASE_HOME=/home/hadoop/hbase_home

export PATH=${HBASE_HOEM}/bin:${PATH}

d) HBase配置

1. ${HBASE_HOME}/conf/hbase-env.sh

############ added by user #########export JAVA_HOME=/home/hadoop/jdk1.7.0_25export HBASE_HOME=/home/hadoop/hbase_homeexport HADOOP_HOME=/home/hadoop/hadoop_homeexport HBASE_LOG_DIR=${HBASE_HOME}/../log/hbaseexport HBASE_MANAGES_ZK=false

2. hbase-site.xml

<configuration>    <property>        <name>hbase.master</name>        <value>rock-hdpnn:49179</value>    </property>    <property>        <name>hbase.master.port</name>        <value>49179</value>    </property>    <property>        <name>hbase.master.info.port</name>        <value>49180</value>    </property>    <property>        <name>hbase.rootdir</name>        <value>hdfs://rock-hdpnn:49175/hbase</value>    </property>    <property>        <name>hbase.tmp.dir</name>        <value>/home/hadoop/hbase_data</value>    </property>    <property>        <name>hbase.cluster.distributed</name>        <value>true</value>    </property>    <property>        <name>hbase.zookeeper.quorum</name>        <value>rock-hdpnn</value>    </property>    <property>        <name>zookeeper.session.timeout</name>        <value>60000</value>    </property>    <property>        <name>hbase.zookeeper.property.clientPort</name>        <value>49178</value>    </property></configuration>

3. regionservers

echo "rock-hdp1" > regionservers

echo "rock-hdp2" >> regionservers
e) 替换hadoop和zookeeper的jar包

ls -al ${HBASE_HOME}/lib中找到hadoop-core-XXXX.jar 和 zookeeper-XXXX.jar

应该把这两个jar包用前面所安装的hadoop和zookeeper下的相对应jar进行替换

f) 启动测试（按顺序：hadoop --> zookeeper --> hbase）

1. 查看进程：jps

2. web查看：http://192.168.15.59:49180 (此端口不是默认的，上述有配置，默认的为60010)

3. 启动hbase shell进行操作：hbase shell