How-to: resolve spark streaming "Not enough space to cache input-0-* in memory! "
来源:互联网 发布:java输入输出语句 编辑:程序博客网 时间:2024/05/17 22:45
Error:
15/09/22 21:52:51 WARN storage.MemoryStore: Failed to reserve initial memory threshold of 1024.0 KB for computing block input-0-1442929965600 in memory.
15/09/22 21:52:51 WARN storage.MemoryStore: Not enough space to cache input-0-1442929965600 in memory! (computed 504.0 B so far)
15/09/22 21:52:51 INFO storage.MemoryStore: Memory use = 4.6 MB (blocks) + 529.8 MB (scratch space shared across 1 thread(s)) = 534.4 MB. Storage limit = 534.5 MB.
15/09/22 21:52:51 WARN storage.MemoryStore: Failed to reserve initial memory threshold of 1024.0 KB for computing block input-0-1442929965600 in memory.
15/09/22 21:52:51 WARN storage.MemoryStore: Not enough space to cache input-0-1442929965600 in memory! (computed 504.0 B so far)
15/09/22 21:52:51 INFO storage.MemoryStore: Memory use = 4.6 MB (blocks) + 529.8 MB (scratch space shared across 1 thread(s)) = 534.4 MB. Storage limit = 534.5 MB.
Root case && resolve:
This is cased by storage MemoryStore is not enough. Two ways to resolve:
- Increase spark.storage.memoryFraction, default is 0.6. Could increase to 0.9.
- Increase spark.driver.memory, my original is 1g, I increase to 2g. Issue resolved.
0 0
- How-to: resolve spark streaming "Not enough space to cache input-0-* in memory! "
- Not enough memory to load specified image
- OGG-02050 Not enough database memory to honor requested MAX_SGA_SIZE
- how-to: resolve crontab does not work
- How to spread receivers over worker hosts in Spark streaming - draft
- How to programmatically clear the filesystem memory cache in C++ on a Linux system?
- How to add swap space in Linux
- how to access file in kernel space
- How to resolve "valid signing identity not found" in provisioning profiles library
- How to Fix Memory Leaks in Java
- How to Clear Inactive Memory in Linux
- How to Profile Memory in Linux
- How to resolve NoClassDefFoundError:
- How to check free space on memory card
- How to input "EOF" signal in unix?
- ZZ How to Flush Memory Cache on Linux Server
- How to convert an input image from one color space to another space
- Error:Unable to start the daemon process: could not reserve enough space for object heap.
- 关于全局变量笔记
- eclipse创建django项目报错误Django not found(新建完成后项目中没有manage、urls等文件)
- centos安装poco-1.6.0-all 报错
- 转-聚类算法总结
- spark安装模式探讨
- How-to: resolve spark streaming "Not enough space to cache input-0-* in memory! "
- 如何选择 Linux 上的跟踪器
- 显示 /隐藏表的列
- Java学习笔记【对象构造】
- android应用程序如何调用支付宝接口
- Mac自带截图快捷键
- Android中如何使用JUnit进行单元测试
- Xcode7下网络连接错误无法联网
- win7 64位安装wireshark,到安装winpcap的时候提示失败,解决办法……