Spark 2.1 , Method used to prevent multiple SparkContexts from being active at the same time
来源:互联网 发布:搞笑文案知乎 编辑:程序博客网 时间:2024/06/06 04:33
SparkContect.scala
// In order to prevent multiple SparkContexts from being active at the same time, mark this // context as having started construction. // NOTE: this must be placed at the beginning of the SparkContext constructor. SparkContext.markPartiallyConstructed(this, allowMultipleContexts)
/** * Called at the beginning of the SparkContext constructor to ensure that no SparkContext is * running. Throws an exception if a running context is detected and logs a warning if another * thread is constructing a SparkContext. This warning is necessary because the current locking * scheme prevents us from reliably distinguishing between cases where another context is being * constructed and cases where another constructor threw an exception. */ private[spark] def markPartiallyConstructed( sc: SparkContext, allowMultipleContexts: Boolean): Unit = { SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized { assertNoOtherContextIsRunning(sc, allowMultipleContexts) contextBeingConstructed = Some(sc) } }
assertNoOtherContextIsRunning
/** * Called to ensure that no other SparkContext is running in this JVM. * * Throws an exception if a running context is detected and logs a warning if another thread is * constructing a SparkContext. This warning is necessary because the current locking scheme * prevents us from reliably distinguishing between cases where another context is being * constructed and cases where another constructor threw an exception. */ private def assertNoOtherContextIsRunning( sc: SparkContext, allowMultipleContexts: Boolean): Unit = { SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized { Option(activeContext.get()).filter(_ ne sc).foreach { ctx => val errMsg = "Only one SparkContext may be running in this JVM (see SPARK-2243)." + " To ignore this error, set spark.driver.allowMultipleContexts = true. " + s"The currently running SparkContext was created at:\n${ctx.creationSite.longForm}" val exception = new SparkException(errMsg) if (allowMultipleContexts) { logWarning("Multiple running SparkContexts detected in the same JVM!", exception) } else { throw exception } } contextBeingConstructed.filter(_ ne sc).foreach { otherContext => // Since otherContext might point to a partially-constructed context, guard against // its creationSite field being null: val otherContextCreationSite = Option(otherContext.creationSite).map(_.longForm).getOrElse("unknown location") val warnMsg = "Another SparkContext is being constructed (or threw an exception in its" + " constructor). This may indicate an error, since only one SparkContext may be" + " running in this JVM (see SPARK-2243)." + s" The other SparkContext was created at:\n$otherContextCreationSite" logWarning(warnMsg) } } }
0 0
- Spark 2.1 , Method used to prevent multiple SparkContexts from being active at the same time
- spark:ValueError: Cannot run multiple SparkContexts at once解决方法
- how to play music and record from the mic at the same time?
- Multiple representations of the same entity are being merged解决方法
- abercrombie france evils. At the same time
- The Method About Identification of the Guest OS Process From VMM At Running Time
- Using MERGE in SQL Server to insert, update and delete at the same time
- Puncture Armstrong viciously vilified truth-tellers at the same time typical doping to make sure you
- only one software management tool is allowed to run at the same time
- How to add css class and id in @Html.TextBox mvc4 at the same time
- Git问题:Cannot update paths and switch to branch 'dev' at the same time.
- fatal: Cannot update paths and switch to branch 'gitlab' at the same time.
- Cannot update paths and switch to branch 'snapshot_mixuse' at the same time.
- Cannot update paths and switch to branch at the same time
- fatal: Cannot update paths and switch to branch 'master' at the same time.
- mysqld_safe Can't log to error log and syslog at the same time.
- How to allow new records to be inserted, but prevent existing data from being edited
- Run-Time Check Failure #3 - The variable 'comstat' is being used without being initialized
- Linux Shell中的${}、##和%%使用范例
- Android中MediaPlayer的常用方法
- 35-并发服务器(多进程)
- CCF NOI1017 价格查询
- Linux shell中运行可执行程序后加上&的作用
- Spark 2.1 , Method used to prevent multiple SparkContexts from being active at the same time
- 旋转数组的最小数字
- centos7下 tomcat8启动慢问题
- 关于UGUI的实例化
- Toorbar的基础使用
- apt update error "E: The package lists or status file could not be parsed or opened."
- 理解AES对称加密
- JS路由
- struts2工作原理