spark 2.1 ConfigProvider

来源：互联网发布：think php 编辑：程序博客网时间：2024/06/09 14:16

ConfigProvider is a trait.

** * A source of configuration values. */private[spark] trait ConfigProvider {  def get(key: String): Option[String]}

EnvProvider gets value form sys.env.

private[spark] class EnvProvider extends ConfigProvider {  override def get(key: String): Option[String] = sys.env.get(key)}

SystemProvider gets value from sys.props.

private[spark] class SystemProvider extends ConfigProvider {  override def get(key: String): Option[String] = sys.props.get(key)}

MapProvider gets value from specified map.

private[spark] class MapProvider(conf: JMap[String, String]) extends ConfigProvider {  override def get(key: String): Option[String] = Option(conf.get(key))}

SparkConfigProvider accepts a map object as constructor parameter.

/** * A config provider that only reads Spark config keys, and considers default values for known * configs when fetching configuration values. */private[spark] class SparkConfigProvider(conf: JMap[String, String]) extends ConfigProvider {  import ConfigEntry._  override def get(key: String): Option[String] = {    if (key.startsWith("spark.")) {      Option(conf.get(key)).orElse(defaultValueString(key))    } else {      None    }  }  private def defaultValueString(key: String): Option[String] = {    findEntry(key) match {      case e: ConfigEntryWithDefault[_] => Option(e.defaultValueString)      case e: ConfigEntryWithDefaultString[_] => Option(e.defaultValueString)      case e: FallbackConfigEntry[_] => get(e.fallback.key)      case _ => None    }  }}

findEntry method is a method of object ConfigEntry.

0 0