spark 错误id意义

可以推广到其他，由系统的错误id类报错找到问题的所在。

1 command exit  code 137  

1 为系统资源不够 ，从节点spark内存超出实际系统内存 系统杀死excutor进程造成的错误

2可以用ulimit -a  看哪个系统限制可以调整  ulimit -v unlimited

2 Too many open files

设置最大文件数限制

 ulimit  -n  124000

/etc/security/limits.conf中soft和hard是限制修改

3 exit code 255 

status of 255 error
错误类型：
java.io.IOException: Task process exit with nonzero status of 255.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:424)

错误原因：
Set mapred.jobtracker.retirejob.interval and mapred.userlog.retain.hours to higher value. By default, their values are 24 hours. These might be the reason for failure, though I'm not sure
参考这里：http://grepalex.com/2012/11/12/hadoop-logging/

http://218.245.3.161/2013/03/31/5965

413/11/07 08:06:17 INFO cluster.ClusterTaskSetManager: Loss was due to org.apache.spark.SparkException
org.apache.spark.SparkException: Error communicating with MapOutputTracker
        at org.apache.spark.MapOutputTracker.askTracker(MapOutputTracker.scala:84)
        at org.apache.spark.MapOutputTracker.getServerStatuses(MapOutputTracker.scala:170)
        at org.apache.spark.BlockStoreShuffleFetcher.fetch(BlockStoreShuffleFetcher.scala:39)
        at org.apache.spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:125)
        at org.apache.spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:116)
        at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
        at scala.collection.immutable.List.foreach(List.scala:76)

源码在 

private[spark] class MapOutputTracker extends Logging {


  private val timeout = Duration.create(System.getProperty("spark.akka.askTimeout", "10").toLong, "seconds")
  
  // Set to the MapOutputTrackerActor living on the driver
  var trackerActor: ActorRef = _


  private var mapStatuses = new TimeStampedHashMap[Int, Array[MapStatus]]


  // Incremented every time a fetch fails so that client nodes know to clear
  // their cache of map output locations if this happens.
  private var epoch: Long = 0
  private val epochLock = new java.lang.Object


  // Cache a serialized version of the output statuses for each shuffle to send them out faster
  var cacheEpoch = epoch
  private val cachedSerializedStatuses = new TimeStampedHashMap[Int, Array[Byte]]


  val metadataCleaner = new MetadataCleaner("MapOutputTracker", this.cleanup)


  // Send a message to the trackerActor and get its result within a default timeout, or
  // throw a SparkException if this fails.
  def askTracker(message: Any): Any = {
    try {
      val future = trackerActor.ask(message)(timeout)
      return Await.result(future, timeout)
    } catch {
      case e: Exception =>
        throw new SparkException("Error communicating with MapOutputTracker", e)
    }
  }

尝试 调大  -Dspark.akka.askTimeout 

4