Spark运行异常：java.lang.AbstractMethodError

异常日志：

17/12/05 18:30:44 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)java.lang.AbstractMethodError: JavaKryoSerializer$1.call(Ljava/lang/Object;)Ljava/lang/Iterable;    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:129)at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:129)    at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:284)at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:171)at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)at org.apache.spark.scheduler.Task.run(Task.scala:89)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)17/12/05 18:30:44 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,ANY, 2249 bytes)17/12/05 18:30:44 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-0,5,main]java.lang.AbstractMethodError: JavaKryoSerializer$1.call(Ljava/lang/Object;)Ljava/lang/Iterable;at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:129)    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:129)at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)    at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:284)    at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:171)    at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)    at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)    at org.apache.spark.scheduler.Task.run(Task.scala:89)    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at java.lang.Thread.run(Thread.java:745)...

异常分析：

这个错误是因为你编译代码时用的Spark 1.*，但是运行时用的Spark 2.*。flatMap的函数签名在这两个版本中是不一样的。

// 在Spark1.*中new FlatMapFunction 重写call方法返回类型为：IterableJavaRDD<String> words = lines.flatMap(new FlatMapFunction<String, String>() {//    @Override    public Iterable<String> call(String line) throws Exception {            return Arrays.asList(line.split("\\\t"));    }});

// 在Spark2.*中new FlatMapFunction 重写call方法返回类型为：IterableJavaRDD<String> words = lines.flatMap(new FlatMapFunction<String, String>() {//    @Override    public Iterator<String> call(String line) throws Exception {            return Arrays.asList(line.split("\\\t")).iterator();    }});