spark提交任务端口占用异常
来源:互联网 发布:js循环while偶数 编辑:程序博客网 时间:2024/06/08 07:09
当在同一台机器上提交多个spark任务时 并且是以client的方式提交,会报端口占用错误
17/05/05 15:51:07 WARN AbstractLifeCycle: FAILED org.spark-project.jetty.server.Server@3c8bdd5b: java.net.BindException: Address already in usejava.net.BindException: Address already in useat sun.nio.ch.Net.bind0(Native Method)at sun.nio.ch.Net.bind(Net.java:433)at sun.nio.ch.Net.bind(Net.java:425)at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)at org.spark-project.jetty.server.Server.doStart(Server.java:293)at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:237)at org.apache.spark.ui.JettyUtils$$anonfun$3.apply(JettyUtils.scala:247)at org.apache.spark.ui.JettyUtils$$anonfun$3.apply(JettyUtils.scala:247)at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1920)at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1911)at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:247)at org.apache.spark.ui.WebUI.bind(WebUI.scala:136)at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:474)at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:474)at scala.Option.foreach(Option.scala:236)at org.apache.spark.SparkContext.<init>(SparkContext.scala:474)at com.fangdd.data.recommender.utils.SparkUtil$.getSparkContext(SparkUtil.scala:25)at com.fangdd.data.recommender.strategy.contentbase.ContentBasedI2URecommenderLoupan$.main(ContentBasedI2URecommenderLoupan.scala:63)at com.fangdd.data.recommender.strategy.contentbase.ContentBasedI2URecommenderLoupan.main(ContentBasedI2URecommenderLoupan.scala)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:497)at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:685)at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Spark任务都会绑定一个端口来显示WebUI,默认端口为4040,如果被占用则依次递增+1端口重试,重试次数由参数spark.port.maxRetries=16控制,默认重试16次后就放弃执行
spark.port.maxRetries
16Maximum number of retries when binding to a port before giving up. When a port is given a specific value (non 0), each subsequent retry will increment the port used in the previous attempt by 1 before retrying. This essentially allows it to try a range of ports from the start port specified to port + maxRetries.初始化参数方式1. 程序代码中初始化SparkConf时,设置conf.set(“spark.port.maxRetries”,“128”)
2. 使用spark-submit提交任务时,--conf spark.port.maxRetries=128
3. 在全局的spark-defaults.conf中添加spark.port.maxRetries 128 , 对所有的application起作用
参考配置文档:https://spark.apache.org/docs/1.6.3/configuration.html
0 0
- spark提交任务端口占用异常
- Spark错误异常-资源占用,任务挂起
- spark 提交任务到spark
- Spark 提交任务详解
- spark任务提交参数
- Spark集群任务提交
- Spark任务的提交
- Spark提交任务到集群
- Spark提交任务到集群
- 提交spark任务时报错
- 【Spark系列6】spark submit提交任务
- spark提交java程序异常
- jvm_bind端口占用异常原因分析
- spark用程序提交任务到yarn
- Spark-submit提交任务到集群
- Spark通过YARN-client提交任务不成功
- Java Web提交任务到Spark
- Spark任务提交jar包依赖解决方案
- 深入java final关键字 基本用法、注意点和优点
- 通过git describe --tags 查询当前branch是从那个tag建立的
- 闭包彻底学习
- vue 数据更新 视图不刷新
- Intellij IDEA怎么像Eclipse一样对Class成员进行排序
- spark提交任务端口占用异常
- 使用org.apache.commons.lang.StringUtils方法containsAny误区 看看你们遇到过没有
- C#里partial关键字的作用(转摘)
- Java的高级编程
- import caffe ImportError: No module named caffe
- RxSwift(3.4.1)
- 灰度发布和AB test
- Hibernate映射普通属性、Hibernate中的各种类型
- Java基础——Object类的三个常用方法