spark livy hue

1.下载 livy
livy为Spark提供了REST接口，有如下特性:
* 提供交互式shell
* 批量提交
* 多用户使用用一个服务(用户模拟)
* 可以从任何地方使用REST的方式提交
* 无需对代码进行任何修改

wget http://archive.cloudera.com/beta/livy/livy-server-0.3.0.zip

配置：

 cat livy-env.sh export SPARK_HOME=/opt/spark-2.1.1-bin-2.6.0-cdh5.11.1export SPARK_CONF_DIR=${SPARK_HOME}/confexport HADOOP_CONF_DIR=/etc/hadoop/conf.cloudera.yarnvim livy.conf#默认是local模式 livy.server.session.factory = yarn

启动：

bin/livy-server17/11/16 14:41:29 INFO StateStore$: Using BlackholeStateStore for recovery.17/11/16 14:41:29 INFO BatchSessionManager: Recovered 0 batch sessions. Next session id: 017/11/16 14:41:29 INFO InteractiveSessionManager: Recovered 0 interactive sessions. Next session id: 017/11/16 14:41:29 INFO InteractiveSessionManager: Heartbeat watchdog thread started.17/11/16 14:41:29 INFO RMProxy: Connecting to ResourceManager at hostname/ip:803217/11/16 14:41:29 INFO WebServer: Starting server on http://ip:8998#放到后台bin/livy-server >/dev/null 2>&1 &

参考：
http://gethue.com/how-to-use-the-livy-spark-rest-job-server-for-interactive-spark-2-2/