问题:Spark执行时0.0.0.0IP错误导致的connect failed
来源:互联网 发布:淘宝捡漏类小说 编辑:程序博客网 时间:2024/05/16 06:57
今天用了虚拟机安装了Spark的伪分布式模式,来体验一下;遇到如下异常:
17/02/21 05:01:17 INFO executor.Executor: Fetching http://0.0.0.28:58292/jars/json-project_2.11-1.0.jar with timestamp 148762447483517/02/21 05:01:18 ERROR executor.Executor: Exception in task 0.0 in stage 0.0 (TID 0)java.net.ConnectException: 无效的参数 (connect failed) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) at sun.net.www.http.HttpClient.New(HttpClient.java:308) at sun.net.www.http.HttpClient.New(HttpClient.java:326) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202) at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966) at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:588) at org.apache.spark.util.Utils$.fetchFile(Utils.scala:394) at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:405) at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:397) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:397) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:193) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)17/02/21 05:01:18 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.net.ConnectException: 无效的参数 (connect failed) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) at sun.net.www.http.HttpClient.New(HttpClient.java:308) at sun.net.www.http.HttpClient.New(HttpClient.java:326) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202) at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966) at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:588) at org.apache.spark.util.Utils$.fetchFile(Utils.scala:394) at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:405) at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:397) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:397) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:193) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
从上一句:
INFO executor.Executor: Fetching http://0.0.0.28:58292/jars/json-project_2.11-1.0.jar with timestamp 1487624474835
且报的异常信息为:java.net.ConnectException: 无效的参数 (connect failed)
为什么会访问一个IP为0.0.0.28的IP呢?本机的IP后缀为192.168.1.28,Spark程序访问本机的IP地址为什么不用127.0.0.1、localhost,又或者是192.168.1.28?
度娘了一把0.0.0.28这样的IP地址与127.0.0.1的区别;知道了0.0.0.0这样的IP代表本机所有的IP,SocketServer为了使client端即可以使用127.0.0.1这样的IP访问,也可以使用localhost访问。但Spark自己的client端发起访问使用0.0.0.0这样的IP,此处应该还可以再优化;
解决办法:
在配置文件spark-env.sh中添加配置:
export SPARK_MASTER_IP=127.0.0.1export SPARK_LOCAL_IP=127.0.0.1
OK,再次重启,解决!
此处记录一下。如有其它解决方案或者该解决办法无法解决该错误,或者本人解释有问题的情况,请留言沟通、斧正。
1 0
- 问题:Spark执行时0.0.0.0IP错误导致的connect failed
- Nginx的connect() failed 错误解决
- 非阻塞connect导致的问题
- 直方图统计导致错误的执行计划
- 直方图统计导致错误的执行计划
- mysqladmin: connect to server at 'localhost' failed初始化密码时的错误处理
- linux下出现connect failed: Permission denied错误的解决办法
- spark集群 Failed to connect to driver!
- mysql版本不同所导致SQL语句执行错误的问题
- 一个多线程问题引发的血案-(代码段执行完毕,子进程未执行完毕导致段错误)
- 如何解决Connect超时导致的阻塞问题
- 错误的转换绑定变量类型导致执行计划错误
- Spark优化-troubleshooting-解决各种序列化导致的错误
- spark troubleshooting--解决各种序列化导致的错误
- 通过端口 1433 连接到主机 localhost 的 TCP/IP 连接失败。错误:“Connection refused: connect的问题
- 通过端口 1433 连接到主机 localhost 的 TCP/IP 连接失败。错误:“Connection refused: connect的问题
- JavaScript执行顺序导致的问题
- 关于Qt执行时Object::connect: No such signal QComboBox::activated()的错误
- 单链表 (面试题)
- CakePHP系列(二)----Bookmarker案例(二)
- C++ 虚析构函数的思考 (Boolan笔记第六周)
- Xcode8制作.a静态库和存放xib和图片的.bundle
- Oracle正确删除archivelog文件
- 问题:Spark执行时0.0.0.0IP错误导致的connect failed
- 欢迎使用CSDN-markdown编辑器
- mysql 新建用户并赋予权限
- CSS三大元素类型:块状元素、内联(行内)元素、内联块状元素
- 筋斗云接口编程
- container_of分析
- 《学习OpenCV》第四章5-a
- PAT 乙级 1007. 素数对猜想 (20) Java版
- C++中继承中的同名成员问题