spark-defaults.conf 里的参数被加载的时机

来源：互联网发布：网络视听发展研究报告编辑：程序博客网时间：2024/05/22 07:01

我在调试Spark参数时，发现里的参数是默认值，把saprk/conf目录加到source也没有用。我查了一年，spark-shell命令执行时只加载spark-env.sh文件，那么spark-defaults.conf里的内容是什么时候加载的。

在org.apache.spark.launcher.CommandBuilderUtils.java 里有spark-defaults.conf的一个常量DEFAULT_PROPERTIES_FILE等于default-defaults.conf

 static final String DEFAULT_PROPERTIES_FILE = "spark-defaults.conf";

在org.apache.spark.launcher.AbstractCommandBuilder.java 里的loadPropertiesFile()里读取DEFAULT_PROPERTIES_FILE文件

/**   * Loads the configuration file for the application, if it exists. This is either the   * user-specified properties file, or the spark-defaults.conf file under the Spark configuration   * directory.   */  private Properties loadPropertiesFile() throws IOException {    Properties props = new Properties();    File propsFile;    if (propertiesFile != null) {      propsFile = new File(propertiesFile);      checkArgument(propsFile.isFile(), "Invalid properties file '%s'.", propertiesFile);    } else {      propsFile = new File(getConfDir(), DEFAULT_PROPERTIES_FILE);    }    if (propsFile.isFile()) {      FileInputStream fd = null;      try {        fd = new FileInputStream(propsFile);        props.load(new InputStreamReader(fd, "UTF-8"));        for (Map.Entry<Object, Object> e : props.entrySet()) {          e.setValue(e.getValue().toString().trim());        }      } finally {        if (fd != null) {          try {            fd.close();          } catch (IOException e) {            // Ignore.          }        }      }    }    return props;  }

然后laodPropertiesFile()方法被getEffectiveConfig方法使用。

Map<String, String> getEffectiveConfig() throws IOException {    if (effectiveConfig == null) {      effectiveConfig = new HashMap<>(conf);      Properties p = loadPropertiesFile();      for (String key : p.stringPropertyNames()) {        if (!effectiveConfig.containsKey(key)) {          effectiveConfig.put(key, p.getProperty(key));        }      }    }    return effectiveConfig;  }

AbstractCommandBuilder 被以下地方使用。

./launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java:class SparkSubmitCommandBuilder extends AbstractCommandBuilder {
./launcher/src/main/java/org/apache/spark/launcher/SparkClassCommandBuilder.java:class SparkClassCommandBuilder extends AbstractCommandBuilder {
./launcher/src/main/java/org/apache/spark/launcher/Main.java: AbstractCommandBuilder builder;

1 0