spark streaming 2.0 RateLimiter

提供waitToPush()方法来限制接收器消费数据的速度。

如果已经非常快地发送了太多消息，waitToPush方法会阻塞当前线程，并且仅在一个新消息被送出后回来，它假定同时只发送一个消息。

park.streaming.receiver.maxRate 限制了每个接收器接受的每秒钟最大的消息数，默认为Long.MaxValue.

spark.streaming.backpressure.initialRate设置为接收器近接受消息的速度，如果不设置，则为park.streaming.receiver.maxRate 。

最后的速度在前两个值取较小的值。

/** * Provides waitToPush() method to limit the rate at which receivers consume data. * * waitToPush method will block the thread if too many messages have been pushed too quickly, * and only return when a new message has been pushed. It assumes that only one message is * pushed at a time. * * The spark configuration spark.streaming.receiver.maxRate gives the maximum number of messages * per second that each receiver will accept. * * @param conf spark configuration */private[receiver] abstract class RateLimiter(conf: SparkConf) extends Logging {  // treated as an upper limit  private val maxRateLimit = conf.getLong("spark.streaming.receiver.maxRate", Long.MaxValue)  private lazy val rateLimiter = GuavaRateLimiter.create(getInitialRateLimit().toDouble)  def waitToPush() {    rateLimiter.acquire()  }  /**   * Return the current rate limit. If no limit has been set so far, it returns {{{Long.MaxValue}}}.   */  def getCurrentLimit: Long = rateLimiter.getRate.toLong  /**   * Set the rate limit to `newRate`. The new rate will not exceed the maximum rate configured by   * {{{spark.streaming.receiver.maxRate}}}, even if `newRate` is higher than that.   *   * @param newRate A new rate in records per second. It has no effect if it's 0 or negative.   */  private[receiver] def updateRate(newRate: Long): Unit =    if (newRate > 0) {      if (maxRateLimit > 0) {        rateLimiter.setRate(newRate.min(maxRateLimit))      } else {        rateLimiter.setRate(newRate)      }    }  /**   * Get the initial rateLimit to initial rateLimiter   */  private def getInitialRateLimit(): Long = {    math.min(conf.getLong("spark.streaming.backpressure.initialRate", maxRateLimit), maxRateLimit)  }}