HBase0.94和Hadoop2 结合

来源:互联网 发布:linux组织文件目录结构 编辑:程序博客网 时间:2024/03/29 01:15

原文地址:https://hbase.apache.org/book.html

4.1.1. Apache HBase 0.94 with Hadoop 2

To get 0.94.x to run on Hadoop 2.2.0, you need to change the hadoop 2 and protobuf versions in the pom.xml: Here is a diff with pom.xml changes:

$ svn diff pom.xmlIndex: pom.xml===================================================================--- pom.xml     (revision 1545157)+++ pom.xml     (working copy)@@ -1034,7 +1034,7 @@     <slf4j.version>1.4.3</slf4j.version>     <log4j.version>1.2.16</log4j.version>     <mockito-all.version>1.8.5</mockito-all.version>-    <protobuf.version>2.4.0a</protobuf.version>+    <protobuf.version>2.5.0</protobuf.version>     <stax-api.version>1.0.1</stax-api.version>     <thrift.version>0.8.0</thrift.version>     <zookeeper.version>3.4.5</zookeeper.version>@@ -2241,7 +2241,7 @@         </property>       </activation>       <properties>-        <hadoop.version>2.0.0-alpha</hadoop.version>+        <hadoop.version>2.2.0</hadoop.version>         <slf4j.version>1.6.1</slf4j.version>       </properties>       <dependencies>


4.1.2. Apache HBase 0.92 and 0.94

HBase 0.92 and 0.94 versions can work with Hadoop versions, 0.20.205, 0.22.x, 1.0.x, and 1.1.x. HBase-0.94 can additionally work with Hadoop-0.23.x and 2.x, but you may have to recompile the code using the specific maven profile (see top level pom.xml)


4.1.3. Apache HBase 0.96

As of Apache HBase 0.96.x, Apache Hadoop 1.0.x at least is required. Hadoop 2 is strongly encouraged (faster but also has fixes that help MTTR). We will no longer run properly on older Hadoops such as 0.20.205 or branch-0.20-append. Do not move to Apache HBase 0.96.x if you cannot upgrade your Hadoop. See HBase, mail # dev - DISCUSS: Have hbase require at least hadoop 1.0.0 in hbase 0.96.0?


4.1.4. Hadoop versions 0.20.x - 1.x

DO NOT use Hadoop versions older than 2.2.0 for HBase versions greater than 1.0. Check release documentation if you are using an older version of HBase for Hadoop related information.


4.1.5. Apache HBase on Secure Hadoop

Apache HBase will run on any Hadoop 0.20.x that incorporates(合并) Hadoop security features as long as you do as suggested above and replace the Hadoop jar that ships with HBase with the secure version. If you want to read more about how to setup Secure HBase, see hbase.secure.configuration.


4.1.6. dfs.datanode.max.transfer.threads

An HDFS DataNode has an upper bound on the number of files that it will serve at any one time. Before doing any loading, make sure you have configured Hadoop’sconf/hdfs-site.xml, setting thedfs.datanode.max.transfer.threads value to at least the following:

<property>  <name>dfs.datanode.max.transfer.threads</name>  <value>4096</value></property>

Be sure to restart your HDFS after making the above configuration.

Not having this configuration in place makes for strange-looking failures. One manifestation is a complaint about missing blocks. For example:

10/12/08 20:10:31 INFO hdfs.DFSClient: Could not obtain block

          blk_XXXXXXXXXXXXXXXXXXXXXX_YYYYYYYY from any node: java.io.IOException: No live nodes

          contain current block. Will get new block locations from namenode and retry...

See also casestudies.max.transfer.threads and note that this property was previously known asdfs.datanode.max.xcievers (e.g.Hadoop HDFS: Deceived by Xciever).

0 0