Spark问题

来源：互联网发布：微信众筹系统源码编辑：程序博客网时间：2024/06/06 12:52
15/07/09 17:50:36 INFO SecurityManager: Changing view acls to: dev001,15/07/09 17:50:36 INFO SecurityManager: Changing modify acls to: dev001,15/07/09 17:50:36 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(dev001, ); users with modify permissions: Set(dev001, )15/07/09 17:50:36 INFO Slf4jLogger: Slf4jLogger startedException in thread "main" org.jboss.netty.channel.ChannelException: Failed to bind to: /172.16.31.163:0at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:391)at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:388)at scala.util.Success$$anonfun$map$1.apply(Try.scala:206)at scala.util.Try$.apply(Try.scala:161)at scala.util.Success.map(Try.scala:206)at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)Caused by: java.net.BindException: 无法指定被请求的地址at sun.nio.ch.Net.bind0(Native Method)at sun.nio.ch.Net.bind(Net.java:444)at sun.nio.ch.Net.bind(Net.java:436)at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)at org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:366)at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:290)at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)at java.lang.Thread.run(Thread.java:745)相同问题：  http://apache-spark-user-list.1001560.n3.nabble.com/org-jboss-netty-channel-ChannelException-Failed-to-bind-to-master-1xx-xx-xx-0-td8400.html  stackoverflow上也有好多类似的解决方法：  1、某位外国人的结局方法   The problem is resolved.I have added SPARK_LOCAL_IP=master in both slaves also.When i changed this my slaves are working.
   Thank you all for your suggestions
    2、好像确实需要把worker和master的spark-env.sh文件配置相同的。如下方式会把worker和master配置完全复制的     见如下：       http://my.oschina.net/132722/blog/196027      然后在修改${SPARK_HOME}/conf下的slaves文件，添加上所有工作节点的主机名或者ip，注意是一行一个，这些工作节点服务器也需要配置ssh无密码登录，并加入master的公匙到工作节点。当然可以不配置，但是每次启动都输入密码很纠结。 
 然后使用scp复制spark文件到其他工作节点。 
     scp -r ${SPARK_HOME} hadoop@主机名:要复制到的目录
 
   然后就可以在master主机${SPARK_HOME}/bin目录下运行start-all.sh启动集群
  


==========================================
这个问题主要是出现在使用master来提交算法时出现的，当使用worker机器提交时不会出现。
也就是说master提交有问题然而worker提交就会出现问题。
0 0