spark java api 调用时报
来源:互联网 发布:手机行车记录软件 编辑:程序博客网 时间:2024/05/22 09:00
java.lang.ClassNotFoundException: JavaWordCount$1
在调用spark给的例子中,我们会碰到提交运行的时候会报空指针问题。那时因为spark集群中找不到你制定的class,所以我们需要手动的将包添加到当前的job中。以JavaWordCount为例,修改后的代码如下:
备注:在运行的参数中需要设置==》spark://localhost:7077 hdfs://localhost:9000/input/test.txt hdfs://localhost:9000/input/result.txt
/** Licensed to the Apache Software Foundation (ASF) under one or more* contributor license agreements. See the NOTICE file distributed with* this work for additional information regarding copyright ownership.* The ASF licenses this file to You under the Apache License, Version 2.0* (the "License"); you may not use this file except in compliance with* the License. You may obtain a copy of the License at** http://www.apache.org/licenses/LICENSE-2.0** Unless required by applicable law or agreed to in writing, software* distributed under the License is distributed on an "AS IS" BASIS,* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.* See the License for the specific language governing permissions and* limitations under the License.*/import java.util.Arrays;import java.util.regex.Pattern;import org.apache.spark.api.java.JavaPairRDD;import org.apache.spark.api.java.JavaRDD;import org.apache.spark.api.java.JavaSparkContext;import org.apache.spark.api.java.function.FlatMapFunction;import org.apache.spark.api.java.function.Function2;import org.apache.spark.api.java.function.PairFunction;import scala.Tuple2;public final class JavaWordCount { private static final Pattern SPACE = Pattern.compile(" "); public static void main(String[] args) throws Exception { if (args.length < 2) { System.err.println("Usage: JavaWordCount <master> <file>"); System.exit(1); } JavaSparkContext ctx = new JavaSparkContext(args[0], "JavaWordCount", System.getenv("SPARK_HOME"), JavaSparkContext.jarOfClass(JavaWordCount.class)); ctx.addJar("/home/hadoop/Desktop/JavaSparkT.jar"); JavaRDD<String> lines = ctx.textFile(args[1], 1); JavaRDD<String> words = lines.flatMap(new FlatMapFunction<String, String>() { @Override public Iterable<String> call(String s) { return Arrays.asList(SPACE.split(s)); } }); JavaPairRDD<String, Integer> ones = words.map(new PairFunction<String, String, Integer>() { @Override public Tuple2<String, Integer> call(String s) { return new Tuple2<String, Integer>(s, 1); } }); JavaPairRDD<String, Integer> counts = ones.reduceByKey(new Function2<Integer, Integer, Integer>() { @Override public Integer call(Integer i1, Integer i2) { return i1 + i2; } }); counts.saveAsTextFile(args[2]);// counts.s /*List<Tuple2<String, Integer>> output = counts.collect(); for (Tuple2<?,?> tuple : output) { System.out.println(tuple._1() + ": " + tuple._2()); }*/ System.exit(0); }}
0 0
- spark java api 调用时报
- 【Spark Java API】broadcast、accumulator
- Spark API Java编程使用方法
- 【Spark Java API】broadcast、accumulator
- Java调用c/c++程序时报错
- Java调用Win API
- java调用windows API
- java调用淘宝api
- scala调用java api
- Java调用Hadoop API
- java调用hive api
- Redis java调用API
- Redis java调用API
- Redis java调用API
- JAVA 调用地图API
- java调用api方法
- Spark基础与Java Api介绍
- Spark基础与Java Api介绍
- 抓包工具抓取本机报文(wiresharke)
- 各种位运算
- savedInstanceState
- Python学习笔记(1)
- Spring的事务处理之编程式事务处理
- spark java api 调用时报
- Activity杀死进程退出
- selector
- 分页读取手机联系人
- C++学习之重载函数
- nodejs 简单爬虫(二)
- URL重写实现session跟踪
- php数据的存储和检索
- 免费无限制的公有云服务Bmob之二 创建表添加行