hive 2.0 的安装（客户端+服务端）

安装hive数据仓库是一件很头疼的事，想来想去根据我的经验决定将我痛苦经历记录一下。
记得去年刚刚学习hadoop的时候尝试安装了一下hive（已经记不清是哪个版本了），做好了配置，输入hive启动时，本以为能启动成功，结果瞅着一大堆异常，看了半天无疾而终，放弃了。
后来跟着视频教程做了一次，没想到竟然成功了，现在想想真该庆幸那时的幸运。
今天又要重新安装一下，还是花费了不少功夫。

今天的安装过程：
<1>选择集群中的一台主机安装mysql （可以任选一台作为hive服务器）
我用的时ubuntu ：

sudo apt install mysql-server

安装过程中需要设置用户名 root 的密码 XXX
<2>安装完成之后：
用root登录创建用户：hive 密码： hive（这里是举例，就是建立一个hive专用用户）
赋予hive用户其他主机登录权限：

grant all on *.* to hive@'%' with grant option;

<3>用hive用户登入mysql，创建数据库hive(数据库名字任意）
<4>安装hive 下载与所安装hadoop相容的hive版本 （我的apache-hive-2.1.1-bin.tar，hadoop是2.7.3版）
解压hive：

sudo tar zvxf apache-hive-2.1.1-bin.tar -C /usr/local

改变目录权限：

sudo chown -R user:user /usr/localapache-hive-2.1.1-bin

(这里的user是你自己的hadoop账户和组啦)
<5> 配置/etc/profile文件 （下面的环境配置我都已自己的配置为准直接粘贴了）

#hive env export HIVE_HOME=/usr/local/apache-hive-2.1.1-binexport PATH=$PATH:${HIVE_HOME}/bin

配置 ${HIVE_HOME}/conf目录下的文件:
Hive配置文件介绍

•hive-site.xml      hive的配置文件•hive-env.sh        hive的运行环境文件•hive-default.xml.template  默认模板•hive-env.sh.template     hive-env.sh默认配置•hive-exec-log4j.properties.template   exec默认配置•hive-log4j.properties.template log默认配置

需要用

cp hive-default.xml.template hive-site.xml                          #默认配置cp hive-env.sh.template hive-env.sh                                 #环境配置文件cp hive-exec-log4j.properties.template hive-exec-log4j.properties   #exec默认配置cp hive-log4j.properties.template hive-log4j.properties             #log默认配置

将这些文件cpoy一下
<6>hive-env.sh (首先 cp hive-env.sh.template hive-env.sh)

    export JAVA_HOME=/usr/local/jvm/jdk1.8.0_111    export HADOOP_HOME=/usr/local/hadoop-2.7.3    export HIVE_HOME=/usr/local/apache-hive-2.1.1-bin    export HIVE_CONF_DIR=$HIVE_HOME/conf

<7>hive-site.xml (首先 cp hive-default.xml.template hive-site.xml)

<property>      <name>javax.jdo.option.ConnectionURL</name>      <value>jdbc:mysql://Goblin01:3306/metastore?createDatabaseIfNotExist=true</value>      <description>the URL of the MySQL database</description>    </property><property>  <name>javax.jdo.option.ConnectionDriverName</name>  <value>com.mysql.jdbc.Driver</value>  <description>Driver class name for a JDBC metastore</description></property><property>  <name>javax.jdo.option.ConnectionUserName</name>  <value>hive</value></property><property>  <name>javax.jdo.option.ConnectionPassword</name>  <value>hive</value></property>

<8>建立存储目录：

hadoop fs -mkdir /tmphadoop fs -mkdir -p /user/hive/warehousehadoop fs -mkdir -p /user/hive/loghadoop fs -chmod g+w /user/hive/warehousehadoop fs -chmod g+w /user/hive/loghadoop fs -chmod g+w /tmp

这里的目录都是在hive-site中配置的，如果跟这个不一样可以修改。
hive默认的内部表存储目录为hdfs：//user/hive/warehouse
<9>将JDBC 驱动 mysql-connect-java-xxx.jar 复制至$HIVE_HOME/lib目录下（可以到网上下载，不必与mysql数据库版本一致）
<10>初始化数据库

schematool -initSchema -dbType mysql

出现以下几行说明初始化成功:

Starting metastore schema initialization to 2.0.0Initialization script hive-schema-2.0.0.mysql.sqlInitialization script completedschemaTool completed

<11>启动：

$  source /etc/profile$  hive

如果出现hive>提示符则说明服务端启动成功（其实服务端也可以作为客户端使用，区别就是服务端需要安装元数据库，而客户端只需安装hive指定服务端地址）
配置完成之后肯定会有一些小异常出现，酌情解决。
这是服务端启动，如果是在其他主机上配置客户端，
需要
<12>将配置好的hive传送至客户端主机相应目录下

$ scp -r $HIVE_HOME username@host:/usr/local/ 

<13>然后在配置一下/etc/profile （安装过程步骤<5>）
<14>hive客户端配置(指定元数据存储的位置）：

<?xml version="1.0" encoding="UTF-8" standalone="no"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><configuration>    <property>       <name>hive.metastore.uris</name>       <value>thrift://hiveServerhost:9083</value>    </property></configuration>

<15>然后在服务端启动metastore即可:

hive --service metastore &

<16>客户端用 $hive 命令启动即可

$ hive

<17>下面列出一些我安装过程中的错误。

1、Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bhive.session.id%7D_resources        at org.apache.hadoop.fs.Path.initialize(Path.java:205)        at org.apache.hadoop.fs.Path.<init>(Path.java:171)        at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:647)        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:563)        at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:531)        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)        at java.lang.reflect.Method.invoke(Method.java:498)        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bhive.session.id%7D_resources        at java.net.URI.checkPath(URI.java:1823)        at java.net.URI.<init>(URI.java:745)        at org.apache.hadoop.fs.Path.initialize(Path.java:202)        ... 12 more

主要看这里 ===> ${system:java.io.tmpdir%7D/$%7Bhive.session.id%7D_resources
像这种类型的异常就是hive-site.xml 中的目录没有配置。
解决办法：
在主机某目录下新建一个文件夹 例如 /home/hive/tmpdir
编辑$HIVE_HOME/conf/hive-site.xml文件找到含有那一串字符的地方替换成刚才建立的目录就行了。

<18>`Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:591)        at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:531)        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)        at java.lang.reflect.Method.invoke(Method.java:498)        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient        at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:226)        at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:366)        at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:310)        at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:290)        at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:266)        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:558)        ... 9 moreCaused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654)        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3367)        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3406)        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3386)        at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3640)        at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:236)        at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:221)        ... 14 moreCaused by: java.lang.reflect.InvocationTargetException        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1652)        ... 23 moreCaused by: MetaException(message:Hive Schema version 2.1.0 does not match metastore's schema version 1.2.0 Metastore is not upgraded or corrupt)        at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7768)        at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7731)        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)        at java.lang.reflect.Method.invoke(Method.java:498)        at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)        at com.sun.proxy.$Proxy21.verifySchema(Unknown Source)        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:565)        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:626)        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:416)        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)        at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6490)        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:238)        at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70)        ... 28 more`

这一大堆看起来很多错误，其实是由于最后一个

Caused by: MetaException(message:Hive Schema version 2.1.0 does not match metastore's schema version 1.2.0 Metastore is not upgraded or corrupt)

说来惭愧，这个问题是由于一开始没有初始化数据库造成的。
引起的，说的是hive的数据库模式和元数据库不匹配，这个问题也是坑呀，我也是第一次遇到，网上搜了半天我只看到schema upgrade 的信息，我想了一下 进入$HIVE_HOME/script/这个目录下看看有什么，果然看到好多数据库，果然看到升级的信息，这些文件都是用来升级的脚本，所以进入mysql，仔细观看了README（根据说明操作），登陆mysql 升级了一下元数据库模式，果然成功了。
解决办法：
进入$HIVE_HOME/scripts/metastore/upgrade/mysql ，根据README文件的说明升级Schema即可。

<19>`Tue Jan 10 20:32:31 CST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.Tue Jan 10 20:32:31 CST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.Tue Jan 10 20:32:31 CST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.Tue Jan 10 20:32:31 CST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set ;?`

这一大堆的warning是新版本SSL验证，先不用管。
<20>客户端连接服务端的时候会出现拒绝连接，这时要关闭服务端防火墙！
当然问题不知止这几个，所以多做经验总结。