HBase配置LZO压缩

HDFS:  hadoop:hadoop-0.20.2-cdh3u0

HBase: HBase:0.90.3 

 

安装过程：

 

1、安装lzo

emerge dev-libs/lzo

 

2.1 下载

download from:

 

Java代码  

wget http://hadoop-gpl-compression.apache-extras.org.codespot.com/files/hadoop-gpl-compression-0.1.0-rc0.tar.gz  

 

2.2 编译

tar -zxvf hadoop-gpl-compression-0.1.0-rc0.tar.gz

cd hadoop-gpl-compression-0.1.0

cp $HADOOP_HOME//hadoop-core-0.20.2-cdh3u0.jar lib/

nano -w src/java/com/hadoop/compression/lzo/LzoCompressor.java

Java代码  

......  

import org.apache.hadoop.conf.Configuration;  

.....  

  

    public void reinit(Configuration conf) {   

      // do nothing   

    }    

 

ant compile-native tar

 

2.3 拷贝需要的文件

cp build/hadoop-gpl-compression-0.1.0-dev.jar $HADOOP_HOME/lib/

cp build/hadoop-gpl-compression-0.1.0-dev.jar $HBASE_HOME/lib/

 

rsync -auPv lib/native $HADOOP_HOME/lib/

rsync -auPv lib/native $HBASE_HOME/lib/

 

3.修改配置重启HBase和Hadoop

vi core-site.xml(添加的部分)

Xml代码  

<property>   

    <name>io.compression.codecs</name>   

 <value>com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec</value>   

    </property>   

  

 <property>   

    <name>io.compression.codec.lzo.class</name>   

    <value>com.hadoop.compression.lzo.LzoCodec</value>   

  </property>     

 

4 测试是否成功

进入hbase目录的/bin/hbase shell里，执行

create 'mytable', {NAME => 'colfam', COMPRESSION => 'lzo'} 

如果提示成功，则表示hbase启用lzo完成。 

 

 

5 参考资料

 

http://wiki.apache.org/hadoop/UsingLzoCompression

http://wiki.apache.org/hadoop/Hbase/HdfsSyncSupport