spark standalone&&yarn模式

来源：互联网发布：电视的mac 编辑：程序博客网时间：2024/04/17 04:01

spark standalone&&yarn模式

standalone模式

1.slaves文件  slave1  slave2  slave32.spark-env.sh    export SCALA_HOME=/usr/local/scala    export JAVA_HOME=/usr/java/default    export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop/bin/hadoop classpath)    export SPARK_MASTER_IP=192.168.137.100    export SPARK_LOCAL_DIRS=/usr/local/spark    export SPARK_WORKER_MEMORY=1g    export SPARK_MASTER_PORT=70773.启动 ./sbin/start-all.sh4.standlone模式之client模式 结果xshell可见 ./bin/spark-submit --class org.apache.spark.examples.SparkPi  --master spark://master:7077  --executor-memory 1G --total-executor-cores 1 ./examples/jars/spark-examples_2.11-2.1.0.jar 100  4.standlone模式之cluster模式 ./bin/spark-submit --class org.apache.spark.examples.SparkPi  --master spark://master:7077 --deploy-mode cluster --supervise --executor-memory 1G --total-executor-cores 1 ./examples/jars/spark-examples_2.11-2.1.0.jar 100

yarn模式

1.spark-env.sh    export SCALA_HOME=/usr/local/scala    export JAVA_HOME=/usr/java/default    export HADOOP_HOME=/usr/local/hadoop    export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop    export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop/bin/hadoop classpath)    export SPARK_MASTER_IP=192.168.137.100    export SPARK_LOCAL_DIRS=/usr/local/spark    export SPARK_WORKER_MEMORY=1g    export SPARK_MASTER_PORT=70772. yarn-client模式：   ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --executor-memory 1G --num-executors 1 ./examples/jars/spark-examples_2.11-2.1.0.jar 1003. yarn-cluster模式：   ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --executor-memory 1G --num-executors 1 ./examples/jars/spark-examples_2.11-2.1.0.jar 100

阅读全文

0 0