Hadoop,Spark ：文件输出压缩

Spark最终结果压缩

sparkContext.textFile(inputPath).flatMap(x => {    TestLog.readFromString(x)}).saveAsTextFile(outputPath, classOf[GzipCodec])
1
2
3

MapReduce最终结果的压缩

FileOutputFormat.setCompressOutput(job, true);//设置压缩FileOutputFormat.setOutputCompressorClass(job, GzipCodec.class);//压缩类型
1
2

Mapper 中间结果压缩

Configuration conf = new Configuration();conf.setBoolean("mapred.compress.map.output",true);conf.setClass("mapred.map.output.compression.codec",GzipCodec.class,CompressionCodec.class);