Spark Streaming+Kafka+Hive+JSON实时增量计算示例

来源：互联网发布：caffe的test iter 编辑：程序博客网时间：2024/06/06 14:08

业务架构：

JavaScript -> Netty -> Kafka -> Spark Streaming + Hive -> Redis -> PHP

1.JavaScript作为统计脚本发送后端服务器

2.Netty用来接收请求，生成用户标识，过滤数据，将原始数据JSON化后写入Kafka

3.Spark Streaming采用Direct Approach (No Receivers)方式从Kafka中获取数据，通过Hive对foreachRDD进行处理，将最终结果写入Redis

4.PHP读取Redis中的统计结果，根据产品需求执行相关推荐业务

部署环境：

CentOS-6.7-x86_64 2.6.32-573.22.1.el6.x86_64

jdk1.8.0_77

spark-1.6.1

scala-2.10.6（spark-1.6.1要求版本）

hadoop-2.7.2（测试时，spark-1.6.1默认为hadoop-2.6.x，之上版本需要自己编译）

kafka_2.10-0.9.0.1（需要对应scala版本）

开发环境：

Windows 10 专业版

myeclipse-2015-stable-1.0

jdk1.7.0_80

节点分布：

192.168.163.141 CoS6-Node1

192.168.163.136 CoS6-Node2

192.168.163.137 CoS6-Node3

1.启动Zookeeper：/opt/zookeeper-3.4.8/bin/zkServer.sh start * 3

2.启动Yarn：/opt/hadoop-2.7.2/sbin/start-all.sh * 1

3.启动ZKFC：/opt/hadoop-2.7.2/sbin/hadoop-daemon.sh start zkfc * 2

4.启动Spark：/opt/spark-1.6.1/sbin/start-all.sh * 1

5.启动Kafka：/opt/kafka_2.10-0.9.0.1/bin/kafka-server-start.sh /opt/kafka_2.10-0.9.0.1/config/server.properties>/data/kafka/run.out & * 3

相关依赖：

<groupId>org.apache.kafka</groupId>

<artifactId>kafka_2.10</artifactId>

</dependency>

<groupId>io.netty</groupId>

<artifactId>netty-all</artifactId>

<version>4.0.36.Final</version>

</dependency>

<groupId>commons-logging</groupId>

<artifactId>commons-logging</artifactId>

</dependency>

<groupId>com.fasterxml.jackson.core</groupId>

<artifactId>jackson-databind</artifactId>

</dependency>

<groupId>org.apache.spark</groupId>

<artifactId>spark-streaming_2.10</artifactId>

</dependency>

<groupId>org.apache.spark</groupId>

<artifactId>spark-hive_2.10</artifactId>

</dependency>

<groupId>org.apache.spark</groupId>

<artifactId>spark-streaming-kafka_2.10</artifactId>

</dependency>

<groupId>redis.clients</groupId>

<artifactId>jedis</artifactId>

</dependency>

</dependencies>

JVM参数：

-Dhadoop.home.dir="D:\\Workspaces\\MyEclipse Professional 2014\\hadoop-common-2.2.0-bin"

-Dhive.exec.scratchdir="C:\\Users\\Ouyang\\AppData\\Local\\Temp\\hive"

-XX:PermSize=128M

-XX:MaxPermSize=4096M

示例代码：

import java.util.Arrays;import java.util.HashMap;import java.util.HashSet;import java.util.Map;import java.util.Set;import kafka.serializer.StringDecoder;import org.apache.spark.SparkConf;import org.apache.spark.api.java.JavaPairRDD;import org.apache.spark.api.java.function.VoidFunction;import org.apache.spark.sql.DataFrame;import org.apache.spark.sql.hive.HiveContext;import org.apache.spark.streaming.Durations;import org.apache.spark.streaming.api.java.JavaPairInputDStream;import org.apache.spark.streaming.api.java.JavaStreamingContext;import org.apache.spark.streaming.kafka.KafkaUtils;public class SparkJson {    public static void main(String[] args) {        Configuration config = Configuration.getInstance();        SparkConf conf = new SparkConf().setMaster(config.getProperty("master")).setAppName(config.getProperty("app.name"));        JavaStreamingContext jssc = new JavaStreamingContext(conf, Durations.milliseconds(500));        final HiveContext sqlContext = new HiveContext(jssc.sparkContext());        Set<String> topicsSet = new HashSet<String>(Arrays.asList(config.getProperty("topic.json")));        Map<String, String> kafkaParams = new HashMap<String, String>();        kafkaParams.put("metadata.broker.list", config.getProperty("metadata.broker.list"));        // Create direct kafka stream with brokers and topics        JavaPairInputDStream<String, String> messages = KafkaUtils.createDirectStream(            jssc, String.class, String.class, StringDecoder.class, StringDecoder.class, kafkaParams, topicsSet        );        // Get the lines, load to sqlContext        messages.foreachRDD(new VoidFunction<JavaPairRDD<String,String>>() {            private static final long serialVersionUID = 1L;            public void call(JavaPairRDD<String, String> t) throws Exception {                if(t.count() < 1) return ;                DataFrame df = sqlContext.read().json(t.values());                df.show();            }        });        // Start the computation        jssc.start();        jssc.awaitTermination();    }}

问题及解决方案：

1.缺少winutils.exe文件

异常内容：

java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:278)

at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:300)

at org.apache.hadoop.util.Shell.<clinit>(Shell.java:293)

at org.apache.hadoop.hive.conf.HiveConf$ConfVars.findHadoopBinary(HiveConf.java:2327)

at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:365)

at org.apache.spark.sql.hive.HiveContext$.newTemporaryConfiguration(HiveContext.scala:730)

at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:215)

at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:208)

at org.apache.spark.sql.hive.HiveContext.functionRegistry$lzycompute(HiveContext.scala:462)

at org.apache.spark.sql.hive.HiveContext.functionRegistry(HiveContext.scala:461)

at org.apache.spark.sql.UDFRegistration.<init>(UDFRegistration.scala:40)

at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:330)

at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:90)

at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101)

at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:103)

at com.leju.esf.cluster.SparkJson.main(SparkJson.java:27)

解决方案：

①下载hadoop-common依照BUILDING.txt进行编译，Windows下依赖zlib和Windows SDK，过程相对麻烦。

项目地址：https://github.com/apache/hadoop-common

②可在github搜索winutils.exe，直接下载别人编码好的文件。

示例地址：https://github.com/srccodes/hadoop-common-2.2.0-bin

③配置HADOOP_HOME，将全部文件拷贝到bin目录下，此处不需要完整安装Hadoop。

④若HADOOP_HOME无效，可在JVM参数中指定：

-Dhadoop.home.dir="D:\\Workspaces\\MyEclipse Professional 2014\\hadoop-common-2.2.0-bin"

2.默认hive.exec.scratchdir目录 /tmp/hive在Windows无法对应到有效的磁盘分区。

异常内容：

Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: ---------

at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)

at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:204)

at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)

at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:218)