python分布式错误解决

来源:互联网 发布:断背山美国影响力知乎 编辑:程序博客网 时间:2024/06/06 14:13

利用python进行MapReduce时,可能会遇到报如下错误:

java.io.IOException: Cannot run program “e_mapper.py”: error=2, No such file or directory

解决办法是用如下命令运行mapper.py和reducer.py:

hadoop jar /home/hdfs/project/hadoop-2.6.1/contrib/hadoop-streaming-2.6.1.jar -D stream.non.zero.exit.is.failure=false -mapper “python emapper.py” -reducer “python ereducer.py” -input /input/* -output /output -file /home/hdfs/emotionanalysis/emapper.py -file /home/hdfs/emotionanalysis/ereducer.py