Spark -6：运行Spark on YARN

翻译：http://spark.apache.org/docs/latest/running-on-yarn.html

支持在YARN上运行（Hadoop NextGen）在Spark0.6.0版本，并在以后的版本中得到改进。

在YARN上启动Spark

确保HADOOP_CONF_DIR或YARN_CONF_DIR指向包含Hadoop集群的（客户端）配置文件的目录。

这些配置用于写入HDFS并连接到YARN ResourceManager。此目录中包含的配置将分发到YARN群集，以便应用程序使用的所有容器都使用相同的配置。如果配置引用了Java系统属性或未由YARN管理的环境变量，则还应在Spark应用程序的配置（驱动程序，执行程序和在客户端模式下运行时的AM）中设置它们。

有两种部署模式可用于在YARN上启动Spark应用程序。在集群模式下，Spark驱动程序在集群上由YARN管理的应用程序主进程内运行，客户端可以在启动应用程序后离开。在客户端模式下，驱动程序在客户端进程中运行，应用程序主服务器仅用于从YARN请求资源。

与Spark独立模式和Mesos模式不同，其中主节点地址在-master参数中指定，在YARN模式下，ResourceManager的地址从Hadoop配置中选取。因此，-master参数是yarn。
在集群模式下启动Spark应用程序：

$ ./bin/spark-submit --class path.to.your.Class --master yarn --deploy-mode cluster [options] <app jar> [app options]

For example:

$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi \    --master yarn \    --deploy-mode cluster \    --driver-memory 4g \    --executor-memory 2g \    --executor-cores 1 \    --queue thequeue \    lib/spark-examples*.jar \    10

以上启动一个YARN客户端程序，启动默认的应用程序主。然后，SparkPi将作为Application Master的子线程运行。客户端将定期轮询应用程序主控以获取状态更新，并在控制台中显示它们。应用程序完成运行后，客户端将退出。请参阅下面的“调试应用程序”部分，了解如何查看驱动程序和执行程序日志。
要在客户端模式下启动Spark应用程序，请执行相同操作，但用客户端替换集群。下面显示了如何在客户端模式下运行spark-shell：

$ ./bin/spark-shell --master yarn --deploy-mode client

添加其他JAR

在集群模式下，驱动程序在与客户端不同的计算机上运行，​​因此SparkContext.addJar将不会开箱即用客户端本地的文件。要使客户端上的文件可用于SparkContext.addJar，请在启动命令中使用--jars选项来包含这些文件。

$ ./bin/spark-submit --class my.main.Class \    --master yarn \    --deploy-mode cluster \    --jars my-other-jar.jar,my-other-other-jar.jar \    my-main-jar.jar \    app_arg1 app_arg2

准备

在YARN上运行Spark需要使用YARN支持构建的Spark。可以从项目网站的下载页面下载。要自己构建Spark，请参考构建Spark。


要使Spark运行时jar可以从YARN端访问，可以指定spark.yarn.archive或spark.yarn.jars。有关详细信息，请参阅Spark属性。如果既没有指定spark.yarn.archive也没有指定spark.yarn.jars，Spark将在$ SPARK_HOME / jars下创建一个包含所有jar的zip文件，并将其上传到分布式缓存。


配置
对于Spark on YARN和其他部署模式，大多数配置相同。有关这些的更多信息，请参阅配置页。这些是特定于YARN上的Spark的配置。

调试应用程序

在YARN术语中，执行者和应用程序masters 在“容器”中运行。 YARN有两种模式用于在应用程序完成后处理容器日志。如果启用日志聚合（使用yarn.log-aggregation-enable配置），容器日志将复制到HDFS并在本地计算机上删除。可以使用yarn logs命令从集群中的任何位置查看这些日志。

yarn logs -applicationId <app ID>

将打印出来自给定应用程序的所有容器的所有日志文件的内容。您还可以使用HDFS shell或API直接在HDFS中查看容器日志文件。可以通过查看您的YARN配置（yarn.nodemanager.remote-app-log-dir和yarn.nodemanager.remote-app-log-dir-suffix）找到它们所在的目录。日志还可以在Spark Web UI的“执行程序”选项卡下找到。您需要同时运行Spark历史记录服务器和MapReduce历史记录服务器，并在yarn-site.xml中正确配置yarn.log.server.url。 Spark历史记录服务器UI上的日志URL将重定向您到MapReduce历史记录服务器以显示聚合日志。

当未打开日志聚合时，日志将在每台计算机上的本地保留在YARN_APP_LOGS_DIR下，通常配置为/ tmp / logs或$ HADOOP_HOME / logs / userlogs，具体取决于Hadoop版本和安装。查看容器的日志需要转到包含它们并在此目录中查找的主机。子目录根据应用程序ID和容器ID组织日志文件。日志还可以在Spark Web UI的“执行程序”选项卡下找到，并且不需要运行MapReduce历史记录服务器。

要查看每个容器的启动环境，请将yarn.nodemanager.delete.debug-delay-sec增加到一个较大的值（例如36000），然后通过yarn.nodemanager.local-dirs访问应用程序缓存，容器所在的节点推出。此目录包含启动脚本，JAR和用于启动每个容器的所有环境变量。这个过程对于调试类路径问题特别有用。 （请注意，启用此功能需要群集设置和所有节点管理器重新启动的管理员权限。因此，这不适用于托管集群）。

要为应用程序主控或执行程序使用自定义log4j配置，请选择以下选项：

1.通过将spark-submit添加到要与应用程序一起上传的文件的--files列表中，上传一个自定义的log4j.properties。

2.添加-Dlog4j.configuration = <配置文件的位置>到spark.driver.extraJavaOptions（对于驱动程序）或spark.executor.extraJavaOptions（对于执行者）。请注意，如果使用文件，则应显式提供文件：协议，并且该文件需要在所有节点上本地存在。

3.更新$ SPARK_CONF_DIR / log4j.properties文件，它将与其他配置一起自动上传。请注意，如果指定了多个选项，其他2个选项的优先级高于此选项。

注意，对于第一个选项，执行器和应用程序主机将共享相同的log4j配置，当它们在同一个节点上运行时（例如，尝试写入同一个日志文件），可能会导致问题。

如果需要引用正确的位置将日志文件放在YARN中，以便YARN可以正确显示和聚合它们，请在log4j.properties中使用spark.yarn.app.container.log.dir。例如，

log4j.appender.file_appender.File=${spark.yarn.app.container.log.dir}/spark.log.

对于流应用程序，配置RollingFileAppender并将文件位置设置为YARN的日志目录将避免由大型日志文件导致的磁盘溢出，并且可以使用YARN的日志实用程序访问日志。

要为应用程序主节点和执行程序使用自定义metrics.properties，请更新$ SPARK_CONF_DIR / metrics.properties文件。它将自动与其他配置一起上传，因此您不需要使用--files手动指定它。

Spark Properties

Property NameDefaultMeaningspark.yarn.am.memory512mAmount of memory to use for the YARN Application Master in client mode, in the same format as JVM memory strings (e.g. 512m, 2g). In cluster mode, use spark.driver.memory instead.

Use lower-case suffixes, e.g. k, m, g, t, and p, for kibi-, mebi-, gibi-, tebi-, and pebibytes, respectively.

spark.yarn.am.cores1Number of cores to use for the YARN Application Master in client mode. In cluster mode, use spark.driver.cores instead.spark.yarn.am.waitTime100sIn cluster mode, time for the YARN Application Master to wait for the SparkContext to be initialized. In client mode, time for the YARN Application Master to wait for the driver to connect to it.spark.yarn.submit.file.replicationThe default HDFS replication (usually 3)HDFS replication level for the files uploaded into HDFS for the application. These include things like the Spark jar, the app jar, and any distributed cache files/archives.spark.yarn.stagingDirCurrent user's home directory in the filesystemStaging directory used while submitting applications.spark.yarn.preserve.staging.filesfalseSet to true to preserve the staged files (Spark jar, app jar, distributed cache files) at the end of the job rather than delete them.spark.yarn.scheduler.heartbeat.interval-ms3000The interval in ms in which the Spark application master heartbeats into the YARN ResourceManager. The value is capped at half the value of YARN's configuration for the expiry interval, i.e. yarn.am.liveness-monitor.expiry-interval-ms.spark.yarn.scheduler.initial-allocation.interval200msThe initial interval in which the Spark application master eagerly heartbeats to the YARN ResourceManager when there are pending container allocation requests. It should be no larger thanspark.yarn.scheduler.heartbeat.interval-ms. The allocation interval will doubled on successive eager heartbeats if pending containers still exist, until spark.yarn.scheduler.heartbeat.interval-ms is reached.spark.yarn.max.executor.failuresnumExecutors * 2, with minimum of 3The maximum number of executor failures before failing the application.spark.yarn.historyServer.address(none)The address of the Spark history server, e.g. host.com:18080. The address should not contain a scheme (http://). Defaults to not being set since the history server is an optional service. This address is given to the YARN ResourceManager when the Spark application finishes to link the application from the ResourceManager UI to the Spark history server UI. For this property, YARN properties can be used as variables, and these are substituted by Spark at runtime. For example, if the Spark history server runs on the same node as the YARN ResourceManager, it can be set to ${hadoopconf-yarn.resourcemanager.hostname}:18080.spark.yarn.dist.archives(none)Comma separated list of archives to be extracted into the working directory of each executor.spark.yarn.dist.files(none)Comma-separated list of files to be placed in the working directory of each executor.spark.yarn.dist.jars(none)Comma-separated list of jars to be placed in the working directory of each executor.spark.executor.instances2The number of executors for static allocation. With spark.dynamicAllocation.enabled, the initial set of executors will be at least this large.spark.yarn.executor.memoryOverheadexecutorMemory * 0.10, with minimum of 384The amount of off-heap memory (in megabytes) to be allocated per executor. This is memory that accounts for things like VM overheads, interned strings, other native overheads, etc. This tends to grow with the executor size (typically 6-10%).spark.yarn.driver.memoryOverheaddriverMemory * 0.10, with minimum of 384The amount of off-heap memory (in megabytes) to be allocated per driver in cluster mode. This is memory that accounts for things like VM overheads, interned strings, other native overheads, etc. This tends to grow with the container size (typically 6-10%).spark.yarn.am.memoryOverheadAM memory * 0.10, with minimum of 384Same as spark.yarn.driver.memoryOverhead, but for the YARN Application Master in client mode.spark.yarn.am.port(random)Port for the YARN Application Master to listen on. In YARN client mode, this is used to communicate between the Spark driver running on a gateway and the YARN Application Master running on YARN. In YARN cluster mode, this is used for the dynamic executor feature, where it handles the kill from the scheduler backend.spark.yarn.queuedefaultThe name of the YARN queue to which the application is submitted.spark.yarn.jars(none)List of libraries containing Spark code to distribute to YARN containers. By default, Spark on YARN will use Spark jars installed locally, but the Spark jars can also be in a world-readable location on HDFS. This allows YARN to cache it on nodes so that it doesn't need to be distributed each time an application runs. To point to jars on HDFS, for example, set this configuration to hdfs:///some/path. Globs are allowed.spark.yarn.archive(none)An archive containing needed Spark jars for distribution to the YARN cache. If set, this configuration replaces spark.yarn.jars and the archive is used in all the application's containers. The archive should contain jar files in its root directory. Like with the previous option, the archive can also be hosted on HDFS to speed up file distribution.spark.yarn.access.namenodes(none)A comma-separated list of secure HDFS namenodes your Spark application is going to access. For example, spark.yarn.access.namenodes=hdfs://nn1.com:8032,hdfs://nn2.com:8032, webhdfs://nn3.com:50070. The Spark application must have access to the namenodes listed and Kerberos must be properly configured to be able to access them (either in the same realm or in a trusted realm). Spark acquires security tokens for each of the namenodes so that the Spark application can access those remote HDFS clusters.spark.yarn.appMasterEnv.[EnvironmentVariableName](none)Add the environment variable specified by EnvironmentVariableName to the Application Master process launched on YARN. The user can specify multiple of these and to set multiple environment variables. In clustermode this controls the environment of the Spark driver and in client mode it only controls the environment of the executor launcher.spark.yarn.containerLauncherMaxThreads25The maximum number of threads to use in the YARN Application Master for launching executor containers.spark.yarn.am.extraJavaOptions(none)A string of extra JVM options to pass to the YARN Application Master in client mode. In cluster mode, use spark.driver.extraJavaOptions instead. Note that it is illegal to set maximum heap size (-Xmx) settings with this option. Maximum heap size settings can be set with spark.yarn.am.memoryspark.yarn.am.extraLibraryPath(none)Set a special library path to use when launching the YARN Application Master in client mode.spark.yarn.maxAppAttemptsyarn.resourcemanager.am.max-attempts in YARNThe maximum number of attempts that will be made to submit the application. It should be no larger than the global number of max attempts in the YARN configuration.spark.yarn.am.attemptFailuresValidityInterval(none)Defines the validity interval for AM failure tracking. If the AM has been running for at least the defined interval, the AM failure count will be reset. This feature is not enabled if not configured, and only supported in Hadoop 2.6+.spark.yarn.executor.failuresValidityInterval(none)Defines the validity interval for executor failure tracking. Executor failures which are older than the validity interval will be ignored.spark.yarn.submit.waitAppCompletiontrueIn YARN cluster mode, controls whether the client waits to exit until the application completes. If set to true, the client process will stay alive reporting the application's status. Otherwise, the client process will exit after submission.spark.yarn.am.nodeLabelExpression(none)A YARN node label expression that restricts the set of nodes AM will be scheduled on. Only versions of YARN greater than or equal to 2.6 support node label expressions, so when running against earlier versions, this property will be ignored.spark.yarn.executor.nodeLabelExpression(none)A YARN node label expression that restricts the set of nodes executors will be scheduled on. Only versions of YARN greater than or equal to 2.6 support node label expressions, so when running against earlier versions, this property will be ignored.spark.yarn.tags(none)Comma-separated list of strings to pass through as YARN application tags appearing in YARN ApplicationReports, which can be used for filtering when querying YARN apps.spark.yarn.keytab(none)The full path to the file that contains the keytab for the principal specified above. This keytab will be copied to the node running the YARN Application Master via the Secure Distributed Cache, for renewing the login tickets and the delegation tokens periodically. (Works also with the "local" master)spark.yarn.principal(none)Principal to be used to login to KDC, while running on secure HDFS. (Works also with the "local" master)spark.yarn.config.gatewayPath(none)A path that is valid on the gateway host (the host where a Spark application is started) but may differ for paths for the same resource in other nodes in the cluster. Coupled with spark.yarn.config.replacementPath, this is used to support clusters with heterogeneous configurations, so that Spark can correctly launch remote processes.

The replacement path normally will contain a reference to some environment variable exported by YARN (and, thus, visible to Spark containers).

For example, if the gateway node has Hadoop libraries installed on /disk1/hadoop, and the location of the Hadoop install is exported by YARN as the HADOOP_HOME environment variable, setting this value to /disk1/hadoop and the replacement path to $HADOOP_HOME will make sure that paths used to launch remote processes properly reference the local YARN configuration.

spark.yarn.config.replacementPath(none)See spark.yarn.config.gatewayPath.spark.yarn.security.credentials.${service}.enabledtrueControls whether to obtain credentials for services when security is enabled. By default, credentials for all supported services are retrieved when those services are configured, but it's possible to disable that behavior if it somehow conflicts with the application being run. For further details please see [Running in a Secure Cluster](running-on-yarn.html#running-in-a-secure-cluster)spark.yarn.rolledLog.includePattern(none)Java Regex to filter the log files which match the defined include pattern and those log files will be aggregated in a rolling fashion. This will be used with YARN's rolling log aggregation, to enable this feature in YARN sideyarn.nodemanager.log-aggregation.roll-monitoring-interval-secondsshould be configured in yarn-site.xml. This feature can only be used with Hadoop 2.6.1+. The Spark log4j appender needs be changed to use FileAppender or another appender that can handle the files being removed while its running. Based on the file name configured in the log4j configuration (like spark.log), the user should set the regex (spark*) to include all the log files that need to be aggregated.spark.yarn.rolledLog.excludePattern(none)Java Regex to filter the log files which match the defined exclude pattern and those log files will not be aggregated in a rolling fashion. If the log file name matches both the include and the exclude pattern, this file will be excluded eventually.

要点：

1.核心请求是否在调度决策中得到执行取决于使用的调度程序及其配置方式。

2.在集群模式下，Spark执行器和Spark驱动程序使用的本地目录将是为YARN（Hadoop YARN配置yarn.nodemanager.local-dirs）配置的本地目录。如果用户指定spark.local.dir，它将被忽略。在客户端模式下，Spark执行程序将使用为YARN配置的本地目录，而Spark驱动程序将使用spark.local.dir中定义的目录。这是因为Spark驱动程序不在客户端模式下在YARN群集上运行，只有Spark执行程序。

3.--files和--archives选项支持使用指定文件名

4.--jars选项允许SparkContext.addJar函数在您使用本地文件并在集群模式下运行时工作。如果您使用HDFS，HTTP，HTTPS或FTP文件，则不需要使用它。

在安全集群中运行

如安全性所述，Kerberos在安全的Hadoop集群中用于验证与服务和客户端相关联的主体。这允许客户端请求这些已验证的服务;向授权的主体授予权利的服务。

Hadoop服务发出hadoop令牌以授予对服务和数据的访问权限。客户端必须首先获取它们将要访问的服务的令牌，并将它们与它们的应用程序一起传递，因为它在YARN集群中启动。

对于Spark应用程序与HDFS，HBase和Hive进行交互，它必须使用启动应用程序的用户的Kerberos凭据获取相关令牌，即身份将成为已启动的Spark应用程序的主体。

这通常在启动时完成：在安全集群中，Spark将自动获取集群的HDFS文件系统的令牌，并可能为HBase和Hive获取。

如果HBase在类路径中，HBase配置声明应用程序是安全的（即hbase-site.xml将hbase.security.authentication设置为kerberos），并且spark.yarn.security.credentials.hbase.enabled将获得HBase令牌未设置为false。

类似地，如果Hive在类路径上，其配置包括“hive.metastore.uris中的元数据存储的URI，并且spark.yarn.security.credentials.hive.enabled未设置为false，则将获得Hive令牌。

如果应用程序需要与其他安全HDFS集群交互，则在启动时必须显式请求访问这些集群所需的令牌。这是通过将它们列在spark.yarn.access.namenodes属性中来实现的。

spark.yarn.access.namenodes hdfs://ireland.example.org:8020/,hdfs://frankfurt.example.org:8020/

Spark通过Java服务机制支持与其他安全感知服务集成（参见java.util.ServiceLoader）。为此，Spark应该可以通过在jar的META-INF / services目录中的相应文件中列出org.apache.spark.deploy.yarn.security.ServiceCredentialProvider的名称来实现org.apache.spark.deploy.yarn.security.ServiceCredentialProvider。可以通过将spark.yarn.security.tokens。{service} .enabled设置为false来禁用这些插件，其中{service}是凭据提供程序的名称。

配置外部Shuffle服务

要在YARN集群中的每个NodeManager上启动Spark Shuffle服务，请按照以下说明操作：

1. 使用YARN配置文件构建Spark。如果使用预打包分发，请跳过此步骤。
2. 找到spark- <version> -yarn-shuffle.jar。如果你自己构建Spark，并且如果你正在使用分布，这应该在$ SPARK_HOME / common / network-yarn / target / scala- <version>下。
3. 将此jar添加到集群中所有NodeManager的类路径
4. 在每个节点上的yarn-site.xml中，将spark_shuffle添加到yarn.nodemanager.aux-services，然后将yarn.nodemanager.aux-services.spark_shuffle.class设置为org.apache.spark.network.yarn.YarnShuffleService。
5. 通过在etc / hadoop / yarn-env.sh中设置YARN_HEAPSIZE（默认为1000）来增加NodeManager的堆大小，以避免shuffle过程中的垃圾回收问题。
6. 重新启动集群中的所有NodeManager。

当在YARN上运行shuffle服务时，以下额外配置选项可用

Property NameDefaultMeaningspark.yarn.shuffle.stopOnFailurefalse当Spark Shuffle Service的初始化失败时是否停止NodeManager。
这可以防止由于SparkShuffle服务未运行的NodeManager上运行容器而导致的应用程序故障。

使用Apache Oozie启动应用程序

Apache Oozie可以作为工作流的一部分启动Spark应用程序。在安全集群中，启动的应用程序将需要相关的令牌来访问集群的服务。如果Spark使用keytab启动，这是自动的。但是，如果Spark将在没有keytab的情况下启动，则设置安全性的责任必须移交给Oozie。有关配置Oozie以获取安全集群和获取作业凭据的详细信息，请参阅Oozie网站上特定版本文档的“身份验证”部分。

For Spark applications, the Oozie workflow must be set up for Oozie to request all tokens which the application needs, including:

• The YARN resource manager.
• The local HDFS filesystem.
• Any remote HDFS filesystems used as a source or destination of I/O.
• Hive —if used.
• HBase —if used.
• The YARN timeline server, if the application interacts with this.

To avoid Spark attempting —and then failing— to obtain Hive, HBase and remote HDFS tokens, the Spark configuration must be set to disable token collection for the services.

The Spark configuration must include the lines:

spark.yarn.security.tokens.hive.enabled   falsespark.yarn.security.tokens.hbase.enabled  false

The configuration option spark.yarn.access.namenodes must be unset.

Kerberos故障排除

调试Hadoop / Kerberos问题可能是“困难的”。一个有用的技术是通过设置HADOOP_JAAS_DEBUG环境变量在Hadoop中启用对Kerberos操作的额外记录。

export HADOOP_JAAS_DEBUG=true

The JDK classes can be configured to enable extra logging of their Kerberos and SPNEGO/REST authentication via the system properties sun.security.krb5.debug and sun.security.spnego.debug=true

-Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true

All these options can be enabled in the Application Master:

spark.yarn.appMasterEnv.HADOOP_JAAS_DEBUG truespark.yarn.am.extraJavaOptions -Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true

Finally, if the log level for org.apache.spark.deploy.yarn.Client is set to DEBUG, the log will include a list of all tokens obtained, and their expiry details