How-to: resolve spark streaming "Not enough space to cache input-0-* in memory! "

来源:互联网 发布:java输入输出语句 编辑:程序博客网 时间:2024/05/17 22:45
Error:
15/09/22 21:52:51 WARN storage.MemoryStore: Failed to reserve initial memory threshold of 1024.0 KB for computing block input-0-1442929965600 in memory.
15/09/22 21:52:51 WARN storage.MemoryStore: Not enough space to cache input-0-1442929965600 in memory! (computed 504.0 B so far)
15/09/22 21:52:51 INFO storage.MemoryStore: Memory use = 4.6 MB (blocks) + 529.8 MB (scratch space shared across 1 thread(s)) = 534.4 MB. Storage limit = 534.5 MB.

Root case && resolve:
This is cased by storage MemoryStore is not enough. Two ways to resolve:
  1. Increase spark.storage.memoryFraction, default is 0.6. Could increase to 0.9.
  2. Increase spark.driver.memory, my original is 1g, I increase to 2g. Issue resolved.
0 0