hadoop中exmaple运行参数分析

问题：

想像hadoop-example...jar那样用hadoop运行jar  中多个类。通过查找资料，好像没有java -cp 那样的参数就是下面问题1中提到的那样运行参数，但是自带的例子却好像可以用这样的方式运行，例如下面，用参数运行不同的class.

hadoop jar hadoop-example*.jar  10 1000000 （好像用投掷飞镖的方式是求pi）

hadoop jar hadoop-example*.jar wordcont intput output  (好像是这么写的一个例子统计单词)

答案：

如何实现？

方法1：

查看examle源代码，发现用的是自带的一个org.apache.hadoop.util.tool 和toolrun

在exampledriver.java中的main中ProgramDriver pgd= new Programdriver()

pgd.addClass("pi",QuasiMonteCarlo.class,QuasiMonteCarlo.DESCRIPTION);

....

pgd.run(argv);////这里应该是根据得到pi 10 10000这样的参数来执行QuasiMonteCarlo.class这个类

方法2:

自己写代码实现上面的功能，也就是类似c++里面经常用的对参数解析的功能，

方法3：

这个是猜测，还没有测试：

http://hadoop.apache.org/docs/current/api/org/apache/hadoop/examples/pi/package-summary.html

Command Line Usages


The command line format is:
$ hadoop org.apache.hadoop.examples.pi.DistBbp \
         <b> <nThreads> <nJobs> <type> <nPart> <remoteDir> <localDir>
And the parameters are:
<b> The number of bits to skip, i.e. compute the (b+1)th position.
<nThreads> The number of working threads.
<nJobs> The number of jobs per sum.
<type> 'm' for map side job, 'r' for reduce side job, 'x' for mix type.
<nPart> The number of parts per job.
<remoteDir> Remote directory for submitting jobs.
<localDir> Local directory for storing output files.

这里指的QuasiMonteCarlo.java里面有个main.通过写包名的方式运行指定的类。当然这里有一句

ToolRunner.run(null,new QuasiMonteCarlo(),argv));这样的代码和方法一的代码统一起来！！！

但是可以猜测也许可以用包名的方式指定运行相应的类中的main

问题1：

未指定Main-Class 的 jar 包如何在运行时指定Main-Class?

正常情况下，java打包成jar包需要在MANIFEST.MF中指定Main-Class项以便运行java -jar ***.jar时找到对应的主类。但是如果我不想在打包成jar包时指定这个类，而是想运行时指定想要运行的类，不知该如何做？

java -jar HelloWorld.jar org.test.HelloWorld

如上方法运行包中的org.test.HelloWorld类时会输出jar中没有主清单属性的错误。当然，我是没有指定Main-Class项，因为我想运行时动态指定。

答案：

-jar的含义就是后面跟的jar包是有main class可独立运行

你应该用-cp / --classpath

java -cp HelloWorld.jar org.test.HelloWorld

问题2：

hadoop Map-Reduce 采用java -jar的方式运行 方式 

答案：

http://blog.163.com/silver9886@126/blog/static/35971862201432163918529/

在hadoop中的教程实例中，大多采用的讲解方式都是Hadoop jar xxx.jar inputfile outputfile 的运行方式，这种方式固然可以，但是，我想怎么样才能直接采用runapplication的形式能够运行jar文件呢？经过两天的研究，终于搞清楚大致流程。

1.需要配置hadoop运行的配置类conf。如果采用了hadoop命令行的运行方式，自然不需要配置conf类。但是因为我们采用java -jar的形式，因此，我们需要配置hadoop的运行参数。具体有以下几种：

conf.set("fs.defaultFS","hdfs://dev65:49100");conf.set("fs.hdfs.impl", "org.apache.hadoop.hdfs.DistributedFileSystem");conf.set("mapreduce.framework.name", "yarn");conf.set("fs.AbstractFileSystem.hdfs.impl", "org.apache.hadoop.fs.Hdfs");

配置之后打包运行后会报错，大致讲的就是map 申请的堆空间是1356M,实际可用是512M.

这其实是我们并没有将map-reduce的map和reduce的最大和最小值配置到conf中，hadoop采用了默认的Map和reduce的堆申请配置，和hadoop实际允许的最大申请值冲突，导致报错。

因此，我们需要将yarn-site.xml 和 mapred-site.xml这两个配置文件放入conf中。这两个配置文件中还有ResourceManager的访问地址和调度地址等配置，当然这些也是必配的。

加入配置后类似这样：

conf.addResource(IOUtils.toInputStream(FileUtils.readFileToString(new File("/usr/local/bigdata/hadoop/conf/yarn-site.xml"), "utf8")));conf.addResource(IOUtils.toInputStream(FileUtils.readFileToString(new File("/usr/local/bigdata/hadoop/conf/mapred-site.xml"), "utf8")));

2.配置完以后如果直接在eclipse中跑是会报错的。提示，找不到类的错误，原因是我们的代码中含有这句话：

job.setJarByClass(MapReduce.class);

这就是给出了MapReduce的class路径，通过这个路径去寻找map和reduce的内部类路径。在eclipse中跑时候是没有打包成class文件的，自然会找不到路径，报类找不到的错误。

3.配置中

 FileInputFormat.addInputPath(Job, new Path("input"));

不支持第一个参数传入JobConf,FileinputFormat 引入的包是

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

其实怀疑引入的是

org.apache.hadoop.mapred.FileInputFormat org.apache.hadoop.mapred.FileOutputFormat

这样我们就可以传入JobConf类型的参数了。

4.将文件打包成jar传入服务器跑java -jar 报找不到内部类

原因是我们采用了eclispe下的fatxx插件来帮助我们打包成jar，采用这种插件打出的包因为被代理掉了，所以jar中的配置，以及内部类都是找不到的，应该采用原始的写MF的方式进行打包

5.这里面网上还说过有其他的问题，包括不建议将map和reduce方法写成单独的类而非写成内部类，这样可能会报找不到类的错误，包括将打成的jar包放在NodeManger 机器上运行，而非resourceManager机器上运行，会报错，因为需要和ResourceManager机器进行通信，ResourceManager机器上没有我们的jar包而会报错(但是我试了在resourceManager上运行没有问题)

6.自己写MF又碰到用eclipse打包时报错line too long错误无法打包。

原因是MF文件Class-Path一行不能太长，否则就会报错。过长可以采用换行，并且换行需要在开头空两个空格。

7.采用MF文件在eclipse下打包后

检查jar包MF文件是空的，原来MF文件要至少有三部分组成，缺一不可：

Manifest-Version: 1.0Main-Class: MapReduceClass-Path: 

之前没有写Manifest-Version这一项

8.打包成功后放入linux服务器还是报找不到类，原因是

Main-Class: MapReduce

e后面有一个空格，这样是不行的。每行的最后是不能有空格的，冒号后面有一个空格，classpath最后有一个空行

8.2 直接打包在本地运行还是会报：

1358)) - Job job_1395919365278_0034 failed with state FAILED due to: Application application_1395919365278_0034 failed 2 times due to AM Container for appattempt_1395919365278_0034_000002 exited with  exitCode: 1 due to: Exception from container-launch:org.apache.hadoop.util.Shell$ExitCodeException: /bin/bash: line 0: fg: no job control        at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)        at org.apache.hadoop.util.Shell.run(Shell.java:379)        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)

怀疑是需要在服务器上运行才行，当然我们也可也下载hadoop2.2 eclispe插件或是网上也有自己写classloader的例子来在本地跑mR代码

9.打包的时候运行会报找不到MainClass，经过检查发现在Mf文件中的classpath中不能有冒号，比如class-path:c:/xxx.jar d:/xxx.jar 因此用linux的目录是没有问题的，但是在windows就会出问题，可以采用相对路径解决，比如lib/xxx.jar...

10.打包之后代码中找不到包中的resource文件（mapred-site.xml和yarn-site.xml），这是因为我们jar包中的文件时不能当做file来读的，new File的话，不能是在jar包中的文件，我们只能将jar包中的文件当做inputstream来读，而不能直接new file出来

11.打包的时候找不到classpath路径下的包，报错，找不到hadoop的类文件。

原因是Mf中配置的classpath也不能是jar包中的路径，因为jar是一个文件了已经，不能从jar包的lib中找到hadoop类文件，需要运行的机器上已有的hadoop包的文件夹配置在classpath中，这就是我们在MF中配置lib/xx.jar 后需要将lib文件夹和jar包放在同一个当前运行的文件夹下的原因。这也引申出为什么需要打成war包的原因，因为只有在war包下，才可以将运行的第三方jar包打入war包中，而且main函数中也可以直接依赖到war包中的三方jar包

12 打包后在服务器端运行程序，成功

付：代码全部以及配置全套

import java.io.File;import java.io.IOException;import java.util.StringTokenizer;import org.apache.commons.io.FileUtils;import org.apache.commons.io.IOUtils;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapred.JobClient;import org.apache.hadoop.mapred.JobConf;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;public class MapReduce {public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable>{  private IntWritable result = new IntWritable();  public void reduce(Text key, Iterable<IntWritable> values, Reducer<Text, IntWritable, Text, IntWritable>.Context context)    throws IOException, InterruptedException  {    int sum = 0;    for (IntWritable val : values) {      sum += val.get();    }    this.result.set(sum);    context.write(key, this.result);  }}public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>{  private static final IntWritable one = new IntWritable(1);  private Text word = new Text();  public void map(Object key, Text value, Mapper<Object, Text, Text, IntWritable>.Context context) throws IOException, InterruptedException  {    StringTokenizer itr = new StringTokenizer(value.toString());    while (itr.hasMoreTokens()) {      this.word.set(itr.nextToken());      context.write(this.word, one);    }  }}      public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException {// TODO Auto-generated method stubConfiguration conf = new Configuration();  conf.set("fs.defaultFS","hdfs://dev65:49100");conf.set("fs.hdfs.impl", "org.apache.hadoop.hdfs.DistributedFileSystem");conf.set("mapreduce.framework.name", "yarn");conf.set("fs.AbstractFileSystem.hdfs.impl", "org.apache.hadoop.fs.Hdfs");conf.set("yarn.resourcemanager.scheduler.address", "dev65:49012");//设置调度地址conf.set("yarn.resourcemanager.address","dev65:49011"); //设置RM 访问位置conf.set("yarn.scheduler.maximum-allocation-mb", "512");//设置调度地址conf.addResource(IOUtils.toInputStream(FileUtils.readFileToString(new File("/usr/local/bigdata/hadoop/conf/yarn-site.xml"), "utf8")));conf.set("mapreduce.map.memory.mb", "256");//设置调度地址conf.addResource(IOUtils.toInputStream(FileUtils.readFileToString(new File("/usr/local/bigdata/hadoop/conf/mapred-site.xml"), "utf8")));
conf.addResource(IOUtils.toInputStream(FileUtils.readFileToString(new File("/usr/local/bigdata/hadoop/conf/core-site.xml"), "utf8")));//JobConf jobConf = new JobConf();// JobClient.runJob(jobConf);Job job = new Job(conf,"sort");job.setJarByClass(MapReduce.class);   job.setMapperClass(TokenizerMapper.class);    job.setCombinerClass(IntSumReducer.class);    job.setReducerClass(IntSumReducer.class);    job.setOutputKeyClass(Text.class);    job.setOutputValueClass(IntWritable.class);    FileInputFormat.addInputPath(job, new Path("input"));    FileOutputFormat.setOutputPath(job, new Path("outputtest"));    job.waitForCompletion(true);        System.out.println("Job ended: ");}}

yarn-site.xml

<?xml version="1.0"?><!--  Licensed under the Apache License, Version 2.0 (the "License");  you may not use this file except in compliance with the License.  You may obtain a copy of the License at    http://www.apache.org/licenses/LICENSE-2.0  Unless required by applicable law or agreed to in writing, software  distributed under the License is distributed on an "AS IS" BASIS,  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the License for the specific language governing permissions and  limitations under the License. See accompanying LICENSE file.--><configuration><property>                <name>yarn.resourcemanager.address</name>                <value>dev65:49011</value>        </property>        <property>                <name>yarn.resourcemanager.scheduler.address</name>                <value>dev65:49012</value>        </property>        <property>                <name>yarn.resourcemanager.resource-tracker.address</name>                <value>dev65:49013</value>        </property>        <property>                <name>yarn.resourcemanager.admin.address</name>                <value>dev65:49014</value>        </property>        <property>                <name>yarn.resourcemanager.webapp.address</name>                <value>dev65:49015</value>        </property>        <property>                <name>yarn.web-proxy.address</name>                <value>dev65:49016</value>        </property>        <property>                <name>yarn.scheduler.minimum-allocation-mb</name>                <value>512</value>        </property>        <property>                <name>yarn.scheduler.maximum-allocation-mb</name>                <value>512</value>        </property>        <property>                <name>yarn.nodemanager.resource.memory-mb</name>                <value>1024</value>        </property>        <property>                <name>yarn.nodemanager.vmem-pmem-ratio</name>                <value>30</value>        </property>        <property>                <name>yarn.nodemanager.local-dirs</name>                <value>/ext/bigdata/hadoop/mapred/local</value>        </property>        <property>                <name>yarn.nodemanager.log-dirs</name>                <value>/ext/bigdata/hadoop/mapred/logs</value>        </property>        <property>                <name>yarn.log-aggregation.retain-seconds</name>                <value>600</value>        </property>        <property>                <name>yarn.app.mapreduce.am.resource.mb</name> <value>64</value>        </property>        <property>                <name>yarn.nodemanager.vmem-check-enabled</name>                <value>false</value>        </property>        <property>                <name>yarn.nodemanager.aux-services</name>                <value>mapreduce_shuffle</value>        </property>                </configuration>

mapred-site.xml

<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!--  Licensed under the Apache License, Version 2.0 (the "License");  you may not use this file except in compliance with the License.  You may obtain a copy of the License at    http://www.apache.org/licenses/LICENSE-2.0  Unless required by applicable law or agreed to in writing, software  distributed under the License is distributed on an "AS IS" BASIS,  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the License for the specific language governing permissions and  limitations under the License. See accompanying LICENSE file.--><!-- Put site-specific property overrides in this file. --><configuration><property><name>mapreduce.framework.name</name><value>yarn</value></property><property><name>mapreduce.map.memory.mb</name><value>256</value></property><property><name>mapreduce.reduce.memory.mb</name><value>256</value></property><property>                <name>mapreduce.map.java.opts</name>                <value>-Xmx128M</value>        </property>        <property>                <name>mapreduce.reduce.java.opts</name>                <value>-Xmx128M</value>        </property><property><name>mapred.child.java.opts</name><value>-Xmx128m</value></property><property><name>mapreduce.task.io.sort.mb</name><value>64</value></property><property>                <name>yarn.app.mapreduce.am.command-opts</name>                <value>-Xmx64m</value>        </property><property><name>mapreduce.task.io.sort.factor</name><value>100</value></property><property><name>mapreduce.reduce.shuffle.parallelcopies</name><value>10</value></property><property><name>mapreduce.jobhistory.address</name><value>dev65:10120</value></property><property><name>mapreduce.jobhistory.webapp.address</name><value>dev65:19988</value></property><property>                <name>mapreduce.jobhistory.intermediate-done-dir</name>                <value>/user/mapred/history/tmp</value>        </property>        <property>                <name>mapreduce.jobhistory.done-dir</name>                <value>/user/mapred/history/done</value>        </property></configuration>

manifes.MF

Manifest-Version: 1.0Main-Class: MapReduceClass-Path: /usr/local/bigdata/hbase/lib/activation-1.1.jar  /usr/local/bigdata/hbase/lib/aopalliance-1.0.jar /usr/local/bigdata/hbase/lib/asm-3.1.jar /usr/local/bigdata/hbase/lib/avro-1.7.4.jar /usr/local/bigdata/hbase/lib/commons-beanutils-1.7.0.jar /usr/local/bigdata/hbase/lib/commons-beanutils-core-1.8.0.jar /usr/local/bigdata/hbase/lib/commons-cli-1.2.jar  /usr/local/bigdata/hbase/lib/commons-codec-1.7.jar /usr/local/bigdata/hbase/lib/commons-collections-3.2.1.jar /usr/local/bigdata/hbase/lib/commons-compress-1.4.1.jar /usr/local/bigdata/hbase/lib/commons-configuration-1.6.jar /usr/local/bigdata/hbase/lib/commons-daemon-1.0.13.jar /usr/local/bigdata/hbase/lib/commons-digester-1.8.jar  /usr/local/bigdata/hbase/lib/commons-el-1.0.jar /usr/local/bigdata/hbase/lib/commons-httpclient-3.1.jar /usr/local/bigdata/hbase/lib/commons-io-2.4.jar /usr/local/bigdata/hbase/lib/commons-lang-2.6.jar /usr/local/bigdata/hbase/lib/commons-logging-1.1.1.jar /usr/local/bigdata/hbase/lib/commons-math-2.2.jar  /usr/local/bigdata/hbase/lib/commons-net-3.1.jar /usr/local/bigdata/hbase/lib/core-3.1.1.jar /usr/local/bigdata/hbase/lib/findbugs-annotations-1.3.9-1.jar /usr/local/bigdata/hbase/lib/gmbal-api-only-3.0.0-b023.jar /usr/local/bigdata/hbase/lib/grizzly-framework-2.1.2.jar /usr/local/bigdata/hbase/lib/grizzly-http-2.1.2.jar  /usr/local/bigdata/hbase/lib/grizzly-http-server-2.1.2.jar /usr/local/bigdata/hbase/lib/grizzly-http-servlet-2.1.2.jar /usr/local/bigdata/hbase/lib/grizzly-rcm-2.1.2.jar /usr/local/bigdata/hbase/lib/guava-12.0.1.jar /usr/local/bigdata/hbase/lib/guice-3.0.jar /usr/local/bigdata/hbase/lib/guice-servlet-3.0.jar  /usr/local/bigdata/hbase/lib/hadoop-annotations-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-auth-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-client-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-common-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-hdfs-2.2.0-tests.jar /usr/local/bigdata/hbase/lib/hadoop-hdfs-2.2.0.jar  /usr/local/bigdata/hbase/lib/hadoop-mapreduce-client-app-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-mapreduce-client-common-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-mapreduce-client-core-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar /usr/local/bigdata/hbase/lib/hadoop-mapreduce-client-jobclient-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-mapreduce-client-shuffle-2.2.0.jar  /usr/local/bigdata/hbase/lib/hadoop-yarn-api-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-yarn-client-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-yarn-common-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-yarn-server-common-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-yarn-server-nodemanager-2.2.0.jar /usr/local/bigdata/hbase/lib/hamcrest-core-1.3.jar  /usr/local/bigdata/hbase/lib/hbase-client-0.96.1.1-hadoop2.jar /usr/local/bigdata/hbase/lib/hbase-common-0.96.1.1-hadoop2-tests.jar /usr/local/bigdata/hbase/lib/hbase-common-0.96.1.1-hadoop2.jar /usr/local/bigdata/hbase/lib/hbase-examples-0.96.1.1-hadoop2.jar /usr/local/bigdata/hbase/lib/hbase-hadoop-compat-0.96.1.1-hadoop2.jar /usr/local/bigdata/hbase/lib/hbase-hadoop2-compat-0.96.1.1-hadoop2.jar  /usr/local/bigdata/hbase/lib/hbase-it-0.96.1.1-hadoop2-tests.jar /usr/local/bigdata/hbase/lib/hbase-it-0.96.1.1-hadoop2.jar /usr/local/bigdata/hbase/lib/hbase-prefix-tree-0.96.1.1-hadoop2.jar /usr/local/bigdata/hbase/lib/hbase-protocol-0.96.1.1-hadoop2.jar /usr/local/bigdata/hbase/lib/hbase-server-0.96.1.1-hadoop2-tests.jar /usr/local/bigdata/hbase/lib/hbase-server-0.96.1.1-hadoop2.jar  /usr/local/bigdata/hbase/lib/hbase-shell-0.96.1.1-hadoop2.jar /usr/local/bigdata/hbase/lib/hbase-testing-util-0.96.1.1-hadoop2.jar /usr/local/bigdata/hbase/lib/hbase-thrift-0.96.1.1-hadoop2.jar /usr/local/bigdata/hbase/lib/high-scale-lib-1.1.1.jar /usr/local/bigdata/hbase/lib/htrace-core-2.01.jar /usr/local/bigdata/hbase/lib/httpclient-4.1.3.jar  /usr/local/bigdata/hbase/lib/httpcore-4.1.3.jar /usr/local/bigdata/hbase/lib/jackson-core-asl-1.8.8.jar /usr/local/bigdata/hbase/lib/jackson-jaxrs-1.8.8.jar /usr/local/bigdata/hbase/lib/jackson-mapper-asl-1.8.8.jar /usr/local/bigdata/hbase/lib/jackson-xc-1.8.8.jar /usr/local/bigdata/hbase/lib/jamon-runtime-2.3.1.jar  /usr/local/bigdata/hbase/lib/jasper-compiler-5.5.23.jar /usr/local/bigdata/hbase/lib/jasper-runtime-5.5.23.jar /usr/local/bigdata/hbase/lib/javax.inject-1.jar /usr/local/bigdata/hbase/lib/javax.servlet-3.1.jar /usr/local/bigdata/hbase/lib/javax.servlet-api-3.0.1.jar /usr/local/bigdata/hbase/lib/jaxb-api-2.2.2.jar  /usr/local/bigdata/hbase/lib/jaxb-impl-2.2.3-1.jar /usr/local/bigdata/hbase/lib/jersey-client-1.9.jar /usr/local/bigdata/hbase/lib/jersey-core-1.8.jar /usr/local/bigdata/hbase/lib/jersey-grizzly2-1.9.jar /usr/local/bigdata/hbase/lib/jersey-guice-1.9.jar /usr/local/bigdata/hbase/lib/jersey-json-1.8.jar  /usr/local/bigdata/hbase/lib/jersey-server-1.8.jar /usr/local/bigdata/hbase/lib/jersey-test-framework-core-1.9.jar /usr/local/bigdata/hbase/lib/jersey-test-framework-grizzly2-1.9.jar /usr/local/bigdata/hbase/lib/jets3t-0.6.1.jar /usr/local/bigdata/hbase/lib/jettison-1.3.1.jar /usr/local/bigdata/hbase/lib/jetty-6.1.26.jar  /usr/local/bigdata/hbase/lib/jetty-sslengine-6.1.26.jar /usr/local/bigdata/hbase/lib/jetty-util-6.1.26.jar /usr/local/bigdata/hbase/lib/jruby-complete-1.6.8.jar /usr/local/bigdata/hbase/lib/jsch-0.1.42.jar /usr/local/bigdata/hbase/lib/jsp-2.1-6.1.14.jar /usr/local/bigdata/hbase/lib/jsp-api-2.1-6.1.14.jar  /usr/local/bigdata/hbase/lib/jsp-api-2.1.jar /usr/local/bigdata/hbase/lib/jsr305-1.3.9.jar /usr/local/bigdata/hbase/lib/junit-4.11.jar /usr/local/bigdata/hbase/lib/libthrift-0.9.0.jar /usr/local/bigdata/hbase/lib/log4j-1.2.17.jar /usr/local/bigdata/hbase/lib/management-api-3.0.0-b012.jar  /usr/local/bigdata/hbase/lib/metrics-core-2.1.2.jar /usr/local/bigdata/hbase/lib/netty-3.6.6.Final.jar /usr/local/bigdata/hbase/lib/paranamer-2.3.jar /usr/local/bigdata/hbase/lib/protobuf-java-2.5.0.jar /usr/local/bigdata/hbase/lib/servlet-api-2.5-6.1.14.jar /usr/local/bigdata/hbase/lib/servlet-api-2.5.jar  /usr/local/bigdata/hbase/lib/slf4j-api-1.6.4.jar /usr/local/bigdata/hbase/lib/slf4j-log4j12-1.6.4.jar /usr/local/bigdata/hbase/lib/snappy-java-1.0.4.1.jar /usr/local/bigdata/hbase/lib/stax-api-1.0.1.jar /usr/local/bigdata/hbase/lib/xmlenc-0.52.jar /usr/local/bigdata/hbase/lib/xz-1.0.jar  /usr/local/bigdata/hbase/lib/zookeeper-3.4.5.jar

这里的包是从hbase中考过来的，可以保证库的可用性

注意 outputfile必须是不存在的文件夹，不然会运行会报错的。不能覆盖之前的job输出结果哦！