Spark 动态上架下架worker

来源：互联网发布：mac ntfs 移动硬盘编辑：程序博客网时间：2024/05/19 12:12

关闭 $SPARK_HOME/sbin/spark-daemon.sh --config conf/ stop org.apache.spark.deploy.worker.Worker 1 --webui-port 8081 spark://llc1:7077开启 $SPARK_HOME/sbin/spark-daemon.sh --config conf/ start org.apache.spark.deploy.worker.Worker 1 --webui-port 8081 -c 24 -m 60G spark://llc1:7077

使用命令

$SPARK_HOME/sbin/spark-daemon.sh [--config <conf-dir>] (start|stop|status) <spark-command> <spark-instance-number> <args...>

第一个参数　：　–config $SPARK_HOME/conf
第二个参数　：　start|stop|status
第三个参数　：　org.apache.spark.deploy.worker.Worker（worker类的路径）
第四个参数　：　这个worker的号码，根据机器上已有的worker数来看
第五个参数　：　启动时的参数，下面是源码解析参数类　WorkerArguments.scala　中截取，都很清楚，传自己需要的参数即可

private def parse(args: List[String]): Unit = args match {  case ("--ip" | "-i") :: value :: tail =>    Utils.checkHost(value, "ip no longer supported, please use hostname " + value)    host = value    parse(tail)  case ("--host" | "-h") :: value :: tail =>    Utils.checkHost(value, "Please use hostname " + value)    host = value    parse(tail)  case ("--port" | "-p") :: IntParam(value) :: tail =>    port = value    parse(tail)  case ("--cores" | "-c") :: IntParam(value) :: tail =>    cores = value    parse(tail)  case ("--memory" | "-m") :: MemoryParam(value) :: tail =>    memory = value    parse(tail)  case ("--work-dir" | "-d") :: value :: tail =>    workDir = value    parse(tail)  case "--webui-port" :: IntParam(value) :: tail =>    webUiPort = value    parse(tail)  case ("--properties-file") :: value :: tail =>    propertiesFile = value    parse(tail)  case ("--help") :: tail =>    printUsageAndExit(0)  case value :: tail =>    if (masters != null) {  // Two positional arguments were given      printUsageAndExit(1)    }    masters = Utils.parseStandaloneMasterUrls(value)    parse(tail)  case Nil =>    if (masters == null) {  // No positional argument was given      printUsageAndExit(1)    }  case _ =>    printUsageAndExit(1)}

一个例子

$SPARK_HOME/sbin/spark-daemon.sh --config conf/ start org.apache.spark.deploy.worker.Worker 1 --webui-port 8082 -c 4 -m 9G spark://llc1:7077

0 0