Spark开发-HA环境的搭建

环境
1、Hadoop2.6
2、spark1.6
3、zookeeper 3.4.6

主机
192.168.18.140 master
192.168.18.141 slave1
192.168.18.142 slave2
192.168.18.143 slave3

现在我们要做的是，在master、slave1、slave2部署zookeeper。在master和slave1上面启动spark的master，刚开始master为active的，slave1位standby的，当我们将master上面的spark进程kill以后，看是否可以切换到slave1上面来

安装
1、zookeeper的安装就不介绍了，详细可以查看文章 http://blog.csdn.net/paicmis/article/details/53264178

2、配置spark的环境信息
修改spark-env.sh文件

export JAVA_HOME=/opt/jdk1.7.0_79export SCALA_HOME=/opt/scala-2.11.4#export SPARK_MASTER_IP=192.168.18.140(主要是注释这段，)export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=192.168.18.140:2181,192.168.18.141:2181,192.168.18.142:2181 -Dspark.deploy.zookeeper.dir=/spark"(这段的意思是告诉spark用zookeeper来协调，指定zookeeper的集群和spark元数据存储的路径)export HADOOP_CONF_DIR=/opt/hadoop-2.6.5/etc/hadoopexport SPARK_WORKER_MEMORY=4gexport SPARK_EXECUTOR_MEMORY=1gexport SPARK_DRIVER_MEMORY=4Gexport SPARK_WORKER_CORES=8

3、配置完成以后需要scp到其他的机器上面去

4、启动 Hadoop zookeeper和spark
主要介绍一下spark的启动过程
先在master上面执行 start-all.sh来启动所有的进程，再在slave1上面启动start-master.sh启动master进程

[root@master sbin]# ./start-all.sh starting org.apache.spark.deploy.master.Master, logging to /opt/spark-1.6.3-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.master.Master-1-master.outslave1: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-1.6.3-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave1.outslave2: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-1.6.3-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave2.outslave3: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-1.6.3-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave3.out[root@slave1 sbin]# ./start-master.sh starting org.apache.spark.deploy.master.Master, logging to /opt/spark-1.6.3-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.master.Master-1-slave1.out[root@slave1 sbin]# 

我们通过http://192.168.18.140:8080/的登录查看是active

通过http://192.168.18.141:8080/查看是standby的

验证
通过下面这个命令我们启动spark-shell来
[root@master bin]# ./spark-shell –master spark://192.168.18.140:7077,192.168.18.141:7077
通过日志我们可以看到这个application向2个master地址发送了请求

我们也可以发现在master上面有这个application进行

现在我们在master机器上面执行下面命令

[root@master sbin]# ./stop-master.sh stopping org.apache.spark.deploy.master.Master

我们可以看到application打印出了如下信息了。master地址变化了

通过http://192.168.18.141:8080/查看是active的，也可以看到application还是running的

我们通过http://192.168.18.140:8080/的登录查看登录不了了

通过zookeeper我们也可以看到如下的信息

[zk: localhost:2181(CONNECTED) 2] ls /spark[master_status, leader_election][zk: localhost:2181(CONNECTED) 3] ls /spark/master_status[app_app-20170920001354-0000, worker_worker-20170808072057-192.168.18.143-60628, worker_worker-20170920000720-192.168.18.142-45588, worker_worker-20170918011207-192.168.18.141-42902][zk: localhost:2181(CONNECTED) 4] ls /spark/leader_election[_c_0e4c8c74-75de-4873-bb6a-13aa8dddc604-latch-0000000001]