hadoop处理lzo格式的压缩文件报错

报错现象：

stderr logs:

  File "<stdin>", line 2

    http://www.mpzw.com/html/120/120905/27272941.html

        ^

SyntaxError: invalid syntax

syslog logs

java.io.IOException: log:null

R/W/S=660/0/0 in:NA [rec/s] out:NA [rec/s]

minRecWrittenToEnableSkip_=9223372036854775807 LOGNAME=null

HOST=null

USER=hadoop

HADOOP_USER=null

last Hadoop input: |null|

last tool output: |null|

Date: Wed Apr 26 20:33:42 CST 2017

java.io.IOException: Broken pipe

at java.io.FileOutputStream.writeBytes(Native Method)

at java.io.FileOutputStream.write(FileOutputStream.java:282)

at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)

at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)

at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)

at java.io.DataOutputStream.write(DataOutputStream.java:90)

at org.apache.hadoop.streaming.io.TextInputWriter.writeUTF8(TextInputWriter.java:72)

at org.apache.hadoop.streaming.io.TextInputWriter.writeValue(TextInputWriter.java:51)

at org.apache.hadoop.streaming.PipeMapper.map(PipeMapper.java:110)

at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)

at org.apache.hadoop.streaming.PipeM

解决办法：

在代码中加入异常处理

1. #!/usr/bin/env python  
2. #coding:utf8  
3. line = line.decode("utf8")  
4. try:  
5. except Exception,ex:  
6.     pass