hive中对lzo压缩文件建立索引实现并行处理

来源：互联网发布：js延时函数编辑：程序博客网时间：2024/05/16 19:06

1,确保创建索引

$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/lib/hadoop-lzo-0.4.10.jar com.hadoop.compression.lzo.DistributedLzoIndexer /user/hive/warehouse/flog

2 如果在hive中新建外部表的语句为

CREATE EXTERNAL TABLE foo (         columnA string,         columnB string )    PARTITIONED BY (date string)    ROW FORMAT DELIMITED FIELDS TERMINATED BY "\t"    STORED AS INPUTFORMAT "com.hadoop.mapred.DeprecatedLzoTextInputFormat"          OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"    LOCATION '/path/to/hive/tables/foo';

3 对于已经存在的表修改语句为

ALTER TABLE foo    SET FILEFORMAT        INPUTFORMAT "com.hadoop.mapred.DeprecatedLzoTextInputFormat"        OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat";

4 alter table后对已经load进表中的数据，需要重新load和创建索引，要不还是不能分块

5 用hadoop streaming编程执行mapreduce作业语句为

hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/contrib/streaming/hadoop-0.20.2-dev-streaming.jar -file /home/pyshell/map.py -file /home/pyshell/red.py -mapper /home/pyshell/map.py -reducer /home/pyshell/red.py -input /aojianlog/20120304/gold/gold_38_3.csv.lzo -output /aojianresult/gold38 -inputformat com.hadoop.mapred.DeprecatedLzoTextInputFormat -jobconf mapred.output.compress=true -jobconf mapred.output.compression.codec=com.hadoop.compression.lzo.LzopCodec

注意如果没有-inputformat com.hadoop.mapred.DeprecatedLzoTextInputFormat 选项的话map作业也不会分片

没有-jobconf mapred.output.compression.codec=com.hadoop.compression.lzo.LzopCodec选项只设置-jobconf mapred.output.compress=true 选项的话 reduce作业输出文件的格式为.lzo_deflate

0 0