Kafka+ Spark Streaming 创建stream编译报错
来源:互联网 发布:单片机按键开关机电路 编辑:程序博客网 时间:2024/09/21 08:14
今天按照spark官网的例子进行stream的运用代码是很简单 但是遇到了两个编译问题让我甚是难受啊
官网给的api为
mvn注入jar包
groupId = org.apache.sparkartifactId = spark-streaming-kafka-0-10_2.11version = 2.2.0代码部分
import org.apache.kafka.clients.consumer.ConsumerRecordimport org.apache.kafka.common.serialization.StringDeserializerimport org.apache.spark.streaming.kafka010._import org.apache.spark.streaming.kafka010.LocationStrategies.PreferConsistentimport org.apache.spark.streaming.kafka010.ConsumerStrategies.Subscribeval kafkaParams = Map[String, Object]( "bootstrap.servers" -> "localhost:9092,anotherhost:9092", "key.deserializer" -> classOf[StringDeserializer], "value.deserializer" -> classOf[StringDeserializer], "group.id" -> "use_a_separate_group_id_for_each_stream", "auto.offset.reset" -> "latest", "enable.auto.commit" -> (false: java.lang.Boolean))val topics = Array("topicA", "topicB")val stream = KafkaUtils.createDirectStream[String, String]( streamingContext, PreferConsistent, Subscribe[String, String](topics, kafkaParams))stream.map(record => (record.key, record.value))
一开始一直报这个错误
Error:scalac: bad symbolic reference. A signature in KafkaUtils.class refers to term serializerin package kafka which is not available.It may be completely missing from the current classpath, or the version onthe classpath might be incompatible with the version used when compiling KafkaUtils.class.
最初认为是这位大神写的 以为缺少jar包,但是jar怎么都搜不到啊
http://blog.csdn.net/wuguobanga/article/details/50780789#reply
后来Google了半小时终于找到原来是我的Scala-sdk 版本和引入的spark-streaming-kafka-0-10_2.11版本不兼容,原先我在IDEA中用的是2.10.6但是这个jar包需要2.11的Scala版本。将idea的Scala改了就行了
2.之后我又出现了一个错误是这样的
Error:(24, 18) Symbol 'type <none>.internal.Logging' is missing from the classpath.This symbol is required by 'object org.apache.spark.streaming.kafka010.KafkaUtils'.Make sure that type Logging is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.A full rebuild may help if 'KafkaUtils.class' was compiled against an incompatible version of <none>.internal. val stream = KafkaUtils.createDirectStream[String, String](
spark1.4版本的spark-assembly-1.4.1-hadoop2.6.0.jar来进行SparkConf的创建
还是版本不兼容
最终将pom.xml改成这样OK了
<dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming-kafka-0-10_2.11</artifactId> <version>2.2.1</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming_2.11</artifactId> <version>2.2.1</version> </dependency>我的代码如下
import org.apache.kafka.common.serialization.StringDeserializerimport org.apache.spark.{SparkConf, TaskContext}import org.apache.spark.streaming.{Seconds, StreamingContext}import org.apache.spark.streaming.kafka010.LocationStrategies.PreferConsistentimport org.apache.spark.streaming.kafka010.ConsumerStrategies.Subscribeimport org.apache.spark.streaming.kafka010.{HasOffsetRanges, KafkaUtils, OffsetRange}object SparkStreaming { def main(args: Array[String]): Unit = { val kafkaParams = Map[String, Object]( "bootstrap.servers" -> "192.168.1.188:9092", "key.deserializer" -> classOf[StringDeserializer], "value.deserializer" -> classOf[StringDeserializer], "group.id" -> "use_a_separate_group_id_for_each_stream", "auto.offset.reset" -> "latest", "enable.auto.commit" -> (false: java.lang.Boolean) ) val conf = new SparkConf().setAppName("Spark Streaming Test").setMaster("local[2]") val ssc = new StreamingContext(conf, Seconds(1)) val topics = Array("new_get_ad") val stream = KafkaUtils.createDirectStream[String, String]( ssc, PreferConsistent, Subscribe[String, String](topics, kafkaParams) ) stream.map(record => (record.key, record.value)) }
阅读全文
0 0
- Kafka+ Spark Streaming 创建stream编译报错
- Spark Streaming 报错:kafka.cluster.BrokerEndPoint cannot be cast to kafka.cluster.Broker
- Spark Streaming整合Kafka
- Spark Streaming + kafka
- Kafka+Spark Streaming集成
- kafka + spark streaming 架构
- spark streaming+kafka
- Spark Streaming + Kafka
- kafka-spark streaming-hbase
- spark streaming 整合kafka
- spark streaming + kafka
- flume kafka spark streaming
- Kafka->Spark Streaming->mongodb
- spark streaming kafka wordcount
- Flume+kafka+spark streaming+scala(sbt编译) 例子演示
- 整合Kafka到Spark Streaming
- Spark Streaming+kafka+eclipse编程
- Spark Streaming 集成 Kafka 总结
- ava 前端加密传输后端解密以及验证码功能
- 图像特征之SURF特征匹配
- 比特币的原理及运作机制
- tensorflow使用gpu进行计算的配置步骤
- Spring的AOP面向切面编程
- Kafka+ Spark Streaming 创建stream编译报错
- 安装tslib中遇到的错误:./autogen.sh: 4: autoreconf: not found
- 国际化i18n
- 主键
- Word2Vec的原理及要点
- Kotlin与Java在语句和表达式上的区别
- 100个房间,100个服务员依次做相反操作,求最后状态
- UE4关于XBox手柄控制源码分析
- 视频转换