HBase配置snappy

来源：互联网发布：工程图纸设计软件编辑：程序博客网时间：2024/06/05 18:24

refer:

http://louishust.github.io/hbase/2015/07/14/snappy-install

编译安装snappy

git clone https://github.com/google/snappy.gityum install -y automake autoconf gcc-c++ cmake libedit./configuremakemake install

编译安装hadoop-snappy

git clone https://github.com/louishust/hadoop-snappy.gitln -s /home/q/java/default/jre/lib/amd64/server/libjvm.so /usr/local/lib/mvn packagecp target/hadoop-snappy-0.0.1-SNAPSHOT.jar /home/q/hadoop/hadoop-2.5.0-cdh5.2.0/lib/cp target/hadoop-snappy-0.0.1-SNAPSHOT-tar/hadoop-snappy-0.0.1-SNAPSHOT/lib/native/Linux-amd64-64/*  /home/q/hadoop/hadoop-2.5.0-cdh5.2.0/lib/nativecp  target/hadoop-snappy-0.0.1-SNAPSHOT-tar/hadoop-snappy-0.0.1-SNAPSHOT/lib/native/Linux-amd64-64/* /home/q/hbase/hbase-0.98.6-cdh5.2.0/lib/native/Linux-amd64-64/

重新编译libhadoop以支持snappy

首先需要protocbuf 2.5.0以上mvn clean package -Pdist,native -DskipTests -Drequire.snappycp  /home/q/hadoop/hadoop-2.5.0-cdh5.2.0/src/hadoop-dist/target/hadoop-2.5.0-cdh5.2.0/lib/native/* /home/q/hadoop/hadoop-2.5.0-cdh5.2.0/lib/native

修改hadoop配置文件

core-site.xml

<property>     <name>io.compression.codecs</name>     <value>org.apache.hadoop.io.compress.SnappyCodec</value></property>

hadoop-env.sh

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/q/hadoop/hadoop-2.5.0-cdh5.2.0/lib/native:/usr/local/lib/export JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:/home/q/hadoop/hadoop-2.5.0-cdh5.2.0/lib/native:/home/q/hadoop/hadoop-2.5.0-cdh5.2.0/lib

yarn-site.xml

<property>     <name>mapreduce.map.output.compress</name>     <value>true</value></property><property>     <name>mapred.map.output.compress.codec</name>     <value>org.apache.hadoop.io.compress.SnappyCodec</value></property>

修改hbase配置文件

hbase-env.sh

### for snappyexport HBASE_HOME=/home/q/hbase/hbase-0.98.6-cdh5.2.0/export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native/:/usr/local/lib/export HBASE_LIBRARY_PATH=$HBASE_LIBRARY_PATH:$HBASE_HOME/lib/native/Linux-amd64-64/:/usr/local/lib/export CLASSPATH=$CLASSPATH:$HBASE_LIBRARY_PATH

配置libjvm

sudo ln -s /home/q/java/default/jre/lib/amd64/server/libjvm.so /usr/local/lib/

重启集群

bin/stop-hbase.shsbin/stop-dfs.shsbin/start-dfs.shbin/start-hbase.sh

验证snappy

touch /tmp/ahbase org.apache.hadoop.hbase.util.CompressionTest /tmp/a snappybin/hbase shellcreate 'tsnappy',{ NAME => 'f', COMPRESSION => 'snappy'}

转载请注明出处：Louis Hust's Blog

阅读全文

0 0