spark deploy_mode配置详解
来源:互联网 发布:淘宝卖家店铺被冻结 编辑:程序博客网 时间:2024/06/06 10:49
spark的deploy_mode可以是client 或cluster
–deploy-mode: Whether to deploy your driver on the worker nodes
(cluster) or locally as an external client (client) (default: client)
注意这里的client和cluster模式的区别在于driver的位置
A common deployment strategy is to submit your application from a
gateway machine that is physically co-located with your worker
machines (e.g. Master node in a standalone EC2 cluster). In this
setup, client mode is appropriate. In client mode, the driver is
launched directly within the spark-submit process which acts as a
client to the cluster. The input and output of the application is
attached to the console. Thus, this mode is especially suitable for
applications that involve the REPL (e.g. Spark shell).In client mode, the driver is launched in the same process as the
client that submits the application. In cluster mode, however, the
driver is launched from one of the Worker processes inside the
cluster, and the client process exits as soon as it fulfills its
responsibility of submitting the application without waiting for the
application to finish.
大意就是,如果你进行spark-submit的机器和worker节点在地理上相近(比如同一个局域网),建议使用client模式,这时driver就在进行spark-submit的线程中。换句话说,进行spark-submit的机器没必要是集群里面的机器,如果你的笔记本安装了Spark环境,就可以提交任务到master节点,这个时候driver就在你的笔记本上运行。这时显然不推荐client模式,因为考虑到网络开销,最好使用cluster模式。使用cluster模式后,Spark就会根据调度算法在集群中选择一个机器作为driver执行的地方。
Master node根本不会执行用户提交的任务,它是负责整个调度的(运行cluster manager)。
- spark deploy_mode配置详解
- Spark submit deploy_mode cluster 中的第三方JAR包
- Spark Metrics配置详解
- Spark Metrics配置详解
- Spark配置参数详解
- spark配置详解
- spark spark_deploy_mode配置详解
- Spark配置属性详解
- Spark配置参数详解
- Spark Metrics配置详解
- flume spark streaming配置详解
- Spark配置属性详解(1)
- Spark配置属性详解(1)
- Spark 性能相关参数配置详解
- spark log4j.properties配置详解与实例
- spark配置IntelliJ开发环境详解
- Hive on Spark安装配置详解
- Spark 性能相关参数配置详解
- springmvc 返回 json
- spoon新手入门教程
- 系统安装之引导剖析,深入理解legacy与uefi
- SRM550
- G
- spark deploy_mode配置详解
- 论文阅读学习
- 子序列
- 关于Select option默认选中及查询后选项值保留的问题
- PhotoShop:对一张图片 批量裁剪
- CCF考试——201703-2学生排队
- 定时,延迟,你造吗!
- nginx安装教程
- C++源文件扩展名