Hadoop Lzo

来源：互联网发布：安知玉如意在线阅读编辑：程序博客网时间：2024/04/30 05:50

解压

在hadoop mapreduce程序中使用如下代码解压lzo文件：

LzopDecompressor lzoDec= new LzopDecompressor(lzoBufferSize);InputStream is = new FileInputStream("xxx");if (is != null) {    LzopInputStream lzoIs = new LzopInputStream(is,lzoDec, lzoBufferSize);    BufferedReader reader = new BufferedReader(new InputStreamReader(lzoIs));    String line = null;    while ((line = reader.readLine()) != null) {        // to do        context.progress();    }    reader.close();    lzoIs.close();    is.close();}

出现了如下的错误：

org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.InternalError: lzo1x_decompress_safe returned: -5    at com.hadoop.compression.lzo.LzoDecompressor.decompressBytesDirect(Native Method)    at com.hadoop.compression.lzo.LzoDecompressor.decompress(LzoDecompressor.java:309)    at com.hadoop.compression.lzo.LzopDecompressor.decompress(LzopDecompressor.java:122)    at com.hadoop.compression.lzo.LzopInputStream.decompress(LzopInputStream.java:247)    at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)

这个主要是lzoBufferSize值设置过小导致的，将值设置大一点就可解决。

0 0