hadoop2.2+hive0.13安装和配置

很久没用过hive了，前几天把hadoop的测试环境全部铲了，需要重新部署，于是下了个hive 0.13. 把过程记录一下：

1. 下载后解压tar包到/hive目录。

2. 进到/hive/conf下面，vi一个hive-site.xml文件，配置内容如下：

<configuration>
        <property>
                <name>javax.jdo.option.ConnectionURL</name>
                <value>jdbc:mysql://h1:3306/hive?createDatabaseIfNotExist=true</value>
        </property>
        <property>
                <name>javax.jdo.option.ConnectionDriverName</name>
                <value>com.mysql.jdbc.Driver</value>
        </property>
        <property>
                <name>javax.jdo.option.ConnectionUserName</name>
                <value>hadoop</value>
        </property>
        <property>
                <name>javax.jdo.option.ConnectionPassword</name>
                <value>123</value>
        </property>
        <property>
                <name>hive.metastore.local</name>
                <value>true</value>
        </property>
        <property>
                <name>hive.metastore.warehouse.dir</name>
                <value>hdfs://h1:9000/hive/warehouse</value>
        </property>
</configuration> 

3. 将mysql的连接器mysql-connector-java-5.1.30-bin.jar拷贝到/hive/lib目录下。

4. 在hdfs上创建/hive/warehouse目录。

5. 此时尝试执行了下/hive/bin/hive命令，提示异常:Access denied for user 'hadoop'@'h1', 这是由于mysql里面没有给hadoop用户，赋权。在mysql中执行：

    grant all privileges on hive.* to hadoop@h1 identified by "123", 再次执行/hive/bin/hive, 异常消失。

6. 此时在执行hive命令时，会出现以下提示：

[hadoop@h1 /hive/conf]$/hive/bin/hive
14/05/02 12:11:37 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/05/02 12:11:37 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
14/05/02 12:11:37 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
14/05/02 12:11:37 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
14/05/02 12:11:37 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
14/05/02 12:11:37 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
14/05/02 12:11:37 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
14/05/02 12:11:37 INFO Configuration.deprecation: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed
14/05/02 12:11:37 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.

上述提示是由HIVE的bug引起的，详情见https://issues.apache.org/jira/browse/HIVE-6159，另外实际上在该0.13中该bug已经修复，但由于官网下载的hive版本是基于hadoop 0.20平台编译的，当前hadoop 2.2环境下需要重新编译hive. 在没重新编译之前，执行show databases;的命令，提示报错如下：

hive> show databases;
OK
Failed with exception java.io.IOException:java.io.IOException: Cannot create an instance of InputFormat class org.apache.hadoop.mapred.TextInputFormat as specified in mapredWork!
Time taken: 0.444 seconds

7. 打开hive的调试模式，运行：hive -hiveconfhive.root.logger=DEBUG,console

可以看到报出的异常如下：

14/05/03 11:24:52 ERROR CliDriver: Failed with exception java.io.IOException:java.io.IOException: Cannot create an instance of InputFormat class org.apache.hadoop.mapred.TextInputFormat as specified in mapredWork!
java.io.IOException: java.io.IOException: Cannot create an instance of InputFormat class org.apache.hadoop.mapred.TextInputFormat as specified in mapredWork!
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:551)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:489)
        at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
        at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1471)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.io.IOException: Cannot create an instance of InputFormat class org.apache.hadoop.mapred.TextInputFormat as specified in mapredWork!
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getInputFormatFromCache(FetchOperator.java:223)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:379)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:515)
        ... 14 more
Caused by: java.lang.RuntimeException: Error in configuring object
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getInputFormatFromCache(FetchOperator.java:219)
        ... 16 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
        ... 19 more
Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
        at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)
        at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
        at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
        ... 24 more
Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1626)
        at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
        ... 26 more

可以看到该报错是由于没有找到lzo的类引起的。

8. 将hadoop-lzo-0.4.15.jar拷贝到/hadoop/lib下面，在hadoop-env.sh中将/hadoop/lib/hadoop-lzo-0.4.15.jar添加到hadoop_classpath中。

再次运行/hive/bin/hive, 不再提示报错。但提示以下：

[hadoop@h1 ~]$/hive/bin/hive
14/05/03 10:53:04 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
14/05/03 10:53:04 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
14/05/03 10:53:04 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
14/05/03 10:53:04 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
14/05/03 10:53:04 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
14/05/03 10:53:04 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/05/03 10:53:04 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative

Logging initialized using configuration in jar:file:/hive/lib/hive-common-0.12.0.jar!/hive-log4j.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/hive/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
hive>

9. 除了以上，为了在hadoop 2.0平台上重新编译hive, 下载源码后，利用mvn编译，执行命令：

#mvn clean install -Phadoop-2,dist提示报错如下：[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-common: Compilation failure: Compilation failure:[ERROR] /hive/hive-src/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1688,30] getHadoopConfNames() is already defined in org.apache.hadoop.hive.conf.HiveConf[ERROR] /hive/hive-src/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1707,30] getHadoopConfNames() is already defined in org.apache.hadoop.hive.conf.HiveConf[ERROR] /hive/hive-src/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1669,3] method does not override or implement a method from a supertype[ERROR] /hive/hive-src/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1687,3] method does not override or implement a method from a supertype[ERROR] /hive/hive-src/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1706,3] method does not override or implement a method from a supertype[ERROR] -> [Help 1]org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-common: Compilation failure        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108)        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:76)        at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:116)        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:361)        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:213)        at org.apache.maven.cli.MavenCli.main(MavenCli.java:157)        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)        at java.lang.reflect.Method.invoke(Method.java:597)        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)Caused by: org.apache.maven.plugin.compiler.CompilationFailureException: Compilation failure        at org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:858)        at org.apache.maven.plugin.compiler.CompilerMojo.execute(CompilerMojo.java:129)        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:133)        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)

查看几个关键类HiveConf.java，ShimLoader, HadoopShims的源码，实在没找到问题出现在哪里，有没有高手指点一二，以上报错什么原因？