spark job server on Spark1.6.0-CDH5.11.0初步使用

1. 下载spark job server（sjs）
   因为下载的是与cdh版本匹配的sjs源码，所以需要在以下连接下载相关版本。
   https://github.com/bjoernlohrmann/spark-jobserver
2. 这次安装使用 yarn模式运行的sjs，具体的安装参照github英文版，参考如下
   https://github.com/spark-jobserver/spark-jobserver
   first of all
   进入sjs源码目录的job-server，修改config目录下的参数。复制local.sh.template为local.sh,以及复制local.conf.template为local.conf。修改local.sh,如下。

 # Environment and deploy file# For use with bin/server_deploy, bin/server_package etc.DEPLOY_HOSTS="master              slave1              slave2"APP_USER=rootAPP_GROUP=root# optional SSH Key to login to deploy server#SSH_KEY=/path/to/keyfile.pemINSTALL_DIR=/usr/local/jsstartLOG_DIR=/var/log/job-serverPIDFILE=spark-jobserver.pidJOBSERVER_MEMORY=1GSPARK_VERSION=1.6.0-cdh5.11.0MAX_DIRECT_MEMORY=1GSPARK_HOME=/opt/cloudera/parcels/CDH/lib/sparkSPARK_CONF_DIR=$SPARK_HOME/conf# Only needed for Mesos deploys#SPARK_EXECUTOR_URI=/home/spark/spark-1.5.0.tar.gz# Only needed for YARN running outside of the cluster# You will need to COPY these files from your cluster to the remote machine# Normally these are kept on the cluster in /etc/hadoop/confYARN_CONF_DIR=/opt/cloudera/parcels/CDH/lib/hadoop/etc/hadoopHADOOP_CONF_DIR=/opt/cloudera/parcels/CDH/lib/hadoop/etc/hadoop## Also optional: extra JVM args for spark-submit# export SPARK_SUBMIT_OPTS+="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5433"SCALA_VERSION=2.10.6

修改的local.conf如下：主要将local模式注释掉，保留yarn-client模式

# spark.master will be passed to each job's JobContext#  master = "local[4]"# master = "mesos://vm28-hulk-pub:5050" master = "yarn-client"

需要将local.sh复制到sjs目录下的bin目录下面，local.conf需要复制到local.sh中设置的INSTALL_DIR=/usr/local/jsstart的目录下
second：编辑好配置文件，检查无误，在sjs的bin目录下
./server_deploy.sh local
这个local是对应刚刚配置的local.sh,local.conf,自此在INSTALL_DIR=/usr/local/jsstart目录下面就会生成文件，有如下：

third:启动
./server_start.sh
检测成功与否：jps查看是否有一个sparksubmit进程，然后在web ui窗口是否起来
fourth：跑测试用例
首先对测试用例进行编译，很简单，只需要进入sbt目录
sbt job-server-tests/package
最后会在job-server-tests的目录下找到测试用例的jar包，进而上传
curl –data-binary @job-server-tests_2.10-0.6.2-KNIME_cdh-5.11.jar 192.168.77.154:8090/jars/test
跑sparkPi的实例
curl -d “input.string = hello job server” ‘192.168.77.151:8090/jobs?appName=test&classPath=spark.jobserver.WordCountExample’
查看job结果
curl -v 192.168.77.154:8090/jobs/a606d18d-5962-4602-814b-73e6dd116e8d
结果也可以在8090端口查看，可以看到在yarn上面就有这么一个job

至此，安装测试完毕，进一步了解使用，编码。