完全分布式安装Hadoop,Hive,Hbase,Hwi,Zookeeper-500行说明

其他部分见：

HDFS YARN zookeeper HBASE HIVE HIVE hwi的启动

HDFS和HBASE动态增加和减少节点

一.下载介质到/opt下面：

1.hadoopwget http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz2.下载jdk1.8.0_1213.下载mysqlwget https://dev.mysql.com/get/Downloads/MySQL-5.7/mysql-community-server-5.7.17-1.el6.x86_64.rpm --no-check-certificate4.下载hivewget http://mirrors.cnnic.cn/apache/hive/hive-2.1.1/apache-hive-2.1.1-bin.tar.gzwget http://mirrors.cnnic.cn/apache/hive/hive-2.1.1/apache-hive-2.1.1-src.tar.gz5.下载zookeeperwget https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.5.2-alpha/zookeeper-3.5.2-alpha.tar.gz   --no-check-certificate6.下载hbasewget https://mirrors.tuna.tsinghua.edu.cn/apache/hbase/1.3.0/hbase-1.3.0-bin.tar.gz --no-check-certificate7.下载其他：#下载storm[root@node02 opt]# wget https://mirrors.cnnic.cn/apache/storm/apache-storm-1.1.0/apache-storm-1.1.0.tar.gz --no-check-certificate#下载sqoopwget https://mirrors.cnnic.cn/apache/sqoop/1.99.7/sqoop-1.99.7-bin-hadoop200.tar.gz --no-check-certificate#下载sparkwget https://mirrors.cnnic.cn/apache/spark/spark-2.1.0/spark-2.1.0-bin-hadoop2.7.tgz --no-check-certificatewget https://mirrors.cnnic.cn/apache/spark/spark-2.1.0/spark-2.1.0-bin-without-hadoop.tgz --no-check-certificate#下载kafkahttps://mirrors.cnnic.cn/apache/kafka/0.10.1.1/kafka_2.11-0.10.1.1.tgz --no-check-certificate#下载flumewget https://mirrors.cnnic.cn/apache/flume/1.7.0/apache-flume-1.7.0-bin.tar.gz --no-check-certificate

二.准备四个节点：

1.node01,node02,node03,node04,而且已经配置了ssh互信，而且已经关闭了防火墙
2.应用解压安装目录为/app
3.HDFS目录为：/app/dirhadoop/

三.安装hadoop

1.环境变量---四个节点都要执行

echo 'export JAVA_HOME=/app/jdk1.8.0_121' >> /etc/profileecho 'export CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar' >> /etc/profileecho 'export HADOOP_HOME=/app/hadoop-2.7.3' >>/etc/profileecho 'export PATH=$PATH:$HADOOP_HOME/bin:$JAVA_HOME/bin:$JAVA_HOME/jre/bin' >> /etc/profilesource /etc/profile

2.创建HDFS使用的目录：--四个节点都要执行

#core-site.xml的hadoop.tmp.dirmkdir -p /app/dirhadoop/tmp#hdfs-site.xml的dfs.namenode.name.dirmkdir -p /app/dirhadoop/dfs/name#hdfs-site.xml的dfs.datanode.data.dirmkdir -p /app/dirhadoop/dfs/data

3.配置hadoop-env.sh

echo 'export JAVA_HOME=/app/jdk1.8.0_121'  >>/app/hadoop-2.7.3/etc/hadoop/hadoop-env.sh

4.配置core-site.xml

(1)hadoop.tmp.dir定义了hdfs的临时文件存放目录(2)>fs.default.name指定了如何访问HDFS，例如hbase设置为：    <name>hbase.rootdir</name><value>hdfs://node01:9000/hbase</value> #vi /app/hadoop-2.7.3/etc/hadoop/core-site.xml<configuration>    <property>      <name>fs.default.name</name>      <value>hdfs://node01:9000</value>    </property>    <property>      <name>hadoop.tmp.dir</name>      <value>/app/dirhadoop/tmp</value>    </property>  </configuration> 

5.配置hdfs-site.xml

(1)dfs.namenode.name.dir(2)dfs.datanode.data.dir<configuration>    <property>      <name>dfs.replication</name>      <value>2</value>    </property>    <property>      <name>dfs.namenode.name.dir</name>      <value>file:/app/dirhadoop/dfs/name</value>    </property>    <property>      <name>dfs.datanode.data.dir</name>      <value>file:/app/dirhadoop/dfs/data</value>    </property>  </configuration> 

6.配置mapred-site.xml

(1)mapreduce.jobhistory.webapp.address定义了访问网址(2)部署完成之后，可以通过sbin/mr-jobhistory-daemon.sh start historyserver启动mapreduce.jobhistory.webapp#vi /app/hadoop-2.7.3/etc/hadoop/mapred-site.xml<configuration>    <property>      <name>mapreduce.framework.name</name>      <value>yarn</value>    </property>    <property>      <name>mapreduce.jobhistory.address</name>      <value>node01:10020</value>    </property>    <property>      <name>mapreduce.jobhistory.webapp.address</name>      <value>node01:19888</value>    </property>  </configuration> 

7.配置yarn-site.xml

(1)#vi /app/hadoop-2.7.3/etc/hadoop/slavesnode02node03node04(2)a.yarn.resourcemanager.webapp.address定义网页访问：/app/hadoop-2.7.3/sbin/start-yarn.sh启动之后，就可以类似http://192.168.13.129:8088/cluster访问了#vi /app/hadoop-2.7.3/etc/hadoop/yarn-site.xml<configuration>   <!-- Site specific YARN configuration properties -->    <property>      <name>yarn.nodemanager.aux-services</name>      <value>mapreduce_shuffle</value>    </property>    <property>      <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>      <value>org.apache.hadoop.mapred.ShuffleHandler</value>    </property>    <property>      <name>yarn.resourcemanager.address</name>      <value>node01:8032</value>    </property>    <property>      <name>yarn.resourcemanager.scheduler.address</name>      <value>node01:8030</value>    </property>    <property>      <name>yarn.resourcemanager.resource-tracker.address</name>      <value>node01:8031</value>    </property>    <property>      <name>yarn.resourcemanager.admin.address</name>      <value>node01:8033</value>    </property>    <property>      <name>yarn.resourcemanager.webapp.address</name>      <value>node01:8088</value>    </property>  </configuration> 

8.node01完成以上配置之后：

scp -r /app/hadoop-2.7.3 node02:/app/scp -r /app/hadoop-2.7.3 node03:/app/scp -r /app/hadoop-2.7.3 node04:/app/

9.在master节点(node01)格式化hdfs

hdfs namenode -format

10.完成之后，master节点启动HDFS

(1)启动hdfs:[root@node01 ~]# /app/hadoop-2.7.3/sbin/start-dfs.shStarting namenodes on [node01]node01: Warning: Permanently added the RSA host key for IP address '192.168.13.129' to the list of known hosts.node01: starting namenode, logging to /app/hadoop-2.7.3/logs/hadoop-root-namenode-node01.outnode02: starting datanode, logging to /app/hadoop-2.7.3/logs/hadoop-root-datanode-node02.outnode03: starting datanode, logging to /app/hadoop-2.7.3/logs/hadoop-root-datanode-node03.outnode04: starting datanode, logging to /app/hadoop-2.7.3/logs/hadoop-root-datanode-node04.outStarting secondary namenodes [0.0.0.0]0.0.0.0: starting secondarynamenode, logging to /app/hadoop-2.7.3/logs/hadoop-root-secondarynamenode-node01.out网页查看：http://192.168.13.129:50070/dfshealth.html#tab-overview(2)启动yarn[root@node01 ~]# /app/hadoop-2.7.3/sbin/start-yarn.shstarting yarn daemonsstarting resourcemanager, logging to /app/hadoop-2.7.3/logs/yarn-root-resourcemanager-node01.outnode03: starting nodemanager, logging to /app/hadoop-2.7.3/logs/yarn-root-nodemanager-node03.outnode02: starting nodemanager, logging to /app/hadoop-2.7.3/logs/yarn-root-nodemanager-node02.outnode04: starting nodemanager, logging to /app/hadoop-2.7.3/logs/yarn-root-nodemanager-node04.out网页查看：<name>yarn.resourcemanager.webapp.address</name>   http://192.168.13.129:8088/cluster/nodes(1)启动historyserversbin/mr-jobhistory-daemon.sh start historyserverhttp://192.168.13.129:19888/jobhistory

四.安装Zookeeper

1.单个创建文件myid

hosts为node01,node02,node03,node04,也可以是IP[root@node01 zookeeper-3.5.2-alpha]# cat myidnode011node022node033node044

2.单个节点配置zoo.cfg

#[root@node01 conf]# cat zoo_sample.cfg#vi zoo.cfgtickTime=2000dataDir=/app/zookeeper-3.5.2-alphaclientPort=2181initLimit=5syncLimit=2server.1=node01:2888:3888 server.2=node02:2888:3888server.3=node03:2888:3888server.4=node04:2888:3888#server.1,server.2,server.3,server.4和myid相匹配

3.4 复制和启动

3.Zookeeper复制到其他所有节点：scp -r /app/zookeeper-3.5.2-alpha node02：/appscp -r /app/zookeeper-3.5.2-alpha node03：/appscp -r /app/zookeeper-3.5.2-alpha node04：/app4.每个节点执行：/app/zookeeper-3.5.2-alpha/bin/zkServer.sh startZooKeeper JMX enabled by defaultUsing config: /app/zookeeper-3.5.2-alpha/bin/../conf/zoo.cfgStarting zookeeper ... STARTED(1)状态：/app/zookeeper-3.5.2-alpha/bin/zkServer.sh statusMode: leaderMode: follower

5.报错处理

[root@node01 logs]# cat zookeeper-root-server-node01.out 2017-04-07 15:47:22,297 [myid:] - INFO  [main:QuorumPeerConfig@116] - Reading configuration from: /app/zookeeper-3.5.2-alpha/bin/../conf/zoo.cfg2017-04-07 15:47:22,304 [myid:] - INFO  [main:QuorumPeerConfig@318] - clientPortAddress is 0.0.0.0/0.0.0.0:21812017-04-07 15:47:22,304 [myid:] - INFO  [main:QuorumPeerConfig@322] - secureClientPort is not set2017-04-07 15:47:22,311 [myid:] - ERROR [main:QuorumPeerMain@86] - Invalid config, exiting abnormallyorg.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Address unresolved: node01:3888 at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.<init>(QuorumPeer.java:242)at org.apache.zookeeper.server.quorum.flexible.QuorumMaj.<init>(QuorumMaj.java:89)at org.apache.zookeeper.server.quorum.QuorumPeerConfig.createQuorumVerifier(QuorumPeerConfig.java:524)at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseDynamicConfig(QuorumPeerConfig.java:557)at org.apache.zookeeper.server.quorum.QuorumPeerConfig.setupQuorumPeerConfig(QuorumPeerConfig.java:530)at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:353)at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:133)at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:110)at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)分析：这个报错原因是zoo.cfg配置文件，server.1=node01:2888:3888后面有空格导致

五.安装HBASE

1.每个节点执行环境变量：

echo 'export HBASE_HOME=/app/hbase-1.3.0' >> /etc/profileecho 'export PATH=$PATH:$HBASE_HOME/bin' >> /etc/profilesource /etc/profile

2.配置hbase-env.sh

(1)mkdir logs #Hbase日志目录(2)echo 'export JAVA_HOME=/app/jdk1.8.0_121' >> conf/hbase-env.shecho 'export HADOOP_HOME=/app/hadoop-2.7.3'>> conf/hbase-env.shecho 'export HBASE_HOME=/app/hbase-1.3.0' >> conf/hbase-env.shecho 'export HBASE_MANAGES_ZK=false ' >> conf/hbase-env.shecho 'export HBASE_LOG_DIR=/app/hbase-1.3.0/logs' >> conf/hbase-env.sh

3.配置hbase-site.xml

(1)hbase.master.info.port指定了hbase web客户端的访问网址(2)hbase.rootdir指定了在HDFS中的位置，根目录参考core-sit.xml的fs.default.name(3)hbase.zookeeper相关属性参照zookeeper的配置：<configuration>    <property>      <name>hbase.rootdir</name>      <value>hdfs://node01:9000/hbase</value>    </property>    <property>       <name>hbase.cluster.distributed</name>       <value>true</value>    </property>    <property>        <name>hbase.master</name>        <value>node01:60000</value>    </property>     <property>      <name>hbase.zookeeper.property.dataDir</name>      <value>/app/zookeeper-3.5.2-alpha</value>    </property>    <property>      <name>hbase.zookeeper.quorum</name>      <value>node01,node02,node03,node04</value>    </property>    <property>      <name>hbase.zookeeper.property.clientPort</name>      <value>2181</value>    </property>   <property>  <name>hbase.master.info.port</name>  <value>60010</value></property> </configuration> 

4.配置regionservers

[root@node01 hbase-1.3.0]# cat conf/regionserversnode02node03node04

5.scp到各个节点:

6.启动

  启动之前得保证ZK和hadoop已经启动  [master@master1 hbase]$ bin/start-hbase.sh

7.报错处理：

(1)hbase(main):006:0> t1 = create 't1', 'f1'ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing(2)是四个节点时间不同步处理：[root@node01 ~]# cat /etc/ntp.confrestrict default nomodifyrestrict 127.0.0.1server 127.127.1.0fudge 127.127.1.0 stratum 10driftfile /var/lib/ntp/driftbroadcastdelay 0.008[root@node02 ~]# ntpdate node01 7 Apr 16:51:55 ntpdate[14684]: step time server 192.168.85.129 offset -38.280736 sec[root@node03 ~]# ntpdate node01 7 Apr 16:52:04 ntpdate[9019]: step time server 192.168.85.129 offset -40.609528 sec[root@node04 ~]# ntpdate node01 7 Apr 16:52:14 ntpdate[9042]: step time server 192.168.85.129 offset -38.668817 sec

六.安装HIVE

1.安装mysql

(1)yum install -y mysql-server mysql mysql-devegroupadd mysqluseradd -g mysql mysqlchown -R mysql:mysql /var/lib/mysql(2)[root@node01 ~]# cat /etc/my.cnf [mysqld]datadir=/var/lib/mysql#socket=/var/lib/mysql/mysql.sockuser=mysql# Disabling symbolic-links is recommended to prevent assorted security riskssymbolic-links=0(3)[root@node01 ~]# service mysqld startStarting mysqld:                                           [  OK  ]

2.配置mysql

#设置密码mysqladmin -u root password 'root'#赋给权限create user 'hive' identified by 'hive'grant all privileges on *.* to 'hive' with grant option;GRANT ALL PRIVILEGES ON *.* TO 'hive'@'node01' IDENTIFIED BY 'hive' WITH GRANT OPTION;GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%' IDENTIFIED BY 'hive' WITH GRANT OPTION;flush privileges;#测试mysql -h node01 -u hive -p 

3.每个节点执行环境变量

echo 'export HIVE_HOME=/app/apache-hive-2.1.1-bin' >> /etc/profileecho 'export PATH=$PATH:$HIVE_HOME/bin' >> /etc/profilesource /etc/profile

4.修改hive-env.sh

cp /app/apache-hive-2.1.1-bin/conf/hive-env.sh.template /app/apache-hive-2.1.1-bin/conf/hive-env.sh/**需要添加**/echo 'export JAVA_HOME=/app/jdk1.8.0_121'  >>/app/apache-hive-2.1.1-bin/conf/hive-env.shln -s $JAVA_HOME/lib/tools.jar $HIVE_HOME/lib/

5.放置mysql驱动

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.41.tar.gzcp mysql-connector-java-5.1.41-bin.jar /app/apache-hive-2.1.1-bin/lib/

6.修改hive-site.xml

1)javax.jdo.option.ConnectionURL安装mysql的节点(2)javax.jdo.option.ConnectionUserName mysql的元数据的用户(3)javax.jdo.option.ConnectionPassword密码#cp hive-default.xml.template hive-site.xml<configuration><property>    <name>javax.jdo.option.ConnectionURL</name>    <value>jdbc:mysql://node01:3306/hive?createDatabaseIfNotExist=true</value>    <description>JDBC connect string for a JDBC metastore</description>  </property>    <property>    <name>javax.jdo.option.ConnectionDriverName</name>    <value>com.mysql.jdbc.Driver</value>    <description>Driver class name for a JDBC metastore</description>  </property>    <property>    <name>javax.jdo.option.ConnectionUserName</name>    <value>hive</value>    <description>username to use against metastore database</description>  </property>    <property>    <name>javax.jdo.option.ConnectionPassword</name>    <value>hive</value>    <description>password to use against metastore database</description>  </property>  </configuration>

7.复制hive到其他节点

scp -r /app/apache-hive-2.1.1-bin/  node02:/app/scp -r /app/apache-hive-2.1.1-bin/  node03:/app/scp -r /app/apache-hive-2.1.1-bin/  node04:/app/

8.创建HDFS目录

hadoop fs -mkdir /hivehadoop fs -mkdir /hive/warehousehadoop fs -chmod g+w /hivehadoop fs -chmod g+w /hive/warehouse

9.初始化元数据

schematool -initSchema -dbType mysql

10.建表测试

create table test_ds  (    id int comment '用户ID',    name string comment '用户名称'  )  comment '测试分区表'  partitioned by(ds string comment '时间分区字段')  clustered by(id) sorted by(name) into 32 buckets      row format delimited   fields terminated by '\t'  stored as rcfile;

七.安装hive hwi

1.打包hive-hwi-2.1.1.war

[root@node01 hwi]# pwd/app/hive_src/apache-hive-2.1.1-srcjar cvfM0 hive-hwi-2.1.1.war -C web/ .cp hive-hwi-2.1.1.war /app/apache-hive-2.1.1-bin/lib/

2.hive-site.xml中添加

<property>    <name>hive.hwi.war.file</name>    <value>lib/hive-hwi-2.1.1.war</value>    <description>This sets the path to the HWI war file, relative to ${HIVE_HOME}. </description></property><property>    <name>hive.hwi.listen.host</name>    <value>node01</value>    <description>This is the host address the Hive Web Interface will listen on</description></property><property>    <name>hive.hwi.listen.port</name>    <value>9999</value>    <description>This is the port the Hive Web Interface will listen on</description></property>

3.启动和查看

[root@node01 lib]# pwd/app/apache-hive-2.1.1-bin/lib[root@node01 lib]# nohup hive --service hwi &查看http://192.168.85.129:9999/hwi/index.jsp

支持原创

公众号：clark_blog
博客：blog.csdn.NET/clark_xu

代码：https://github.com/clark99