several technical limits to the NameNode
来源:互联网 发布:java工控软件开发 编辑:程序博客网 时间:2024/06/03 15:58
There are several technical limits to the NameNode (NN), and facing any of them will limit your scalability.
1. Memory. NN consume about 150 bytes per each block. From here you can calculate how much RAM you need for your data. There is good discussion: Namenode file quantity limit
2. IO. NN is doing 1 IO for each change to filesystem (like create, delete block etc). So your local IO should allow enough. It is harder to estimate how much you need. Taking into account fact that we are limited in number of blocks by memory you will not claim this limit unless your cluster is very big. If it is - consider SSD.
3. CPU. Namenode has considerable load keeping track of health of all blocks on all datanodes. Each datanode once a period of time report state of all its block. Again, unless cluster is not too big it should not be a problem.
1. Memory. NN consume about 150 bytes per each block. From here you can calculate how much RAM you need for your data. There is good discussion: Namenode file quantity limit
2. IO. NN is doing 1 IO for each change to filesystem (like create, delete block etc). So your local IO should allow enough. It is harder to estimate how much you need. Taking into account fact that we are limited in number of blocks by memory you will not claim this limit unless your cluster is very big. If it is - consider SSD.
3. CPU. Namenode has considerable load keeping track of health of all blocks on all datanodes. Each datanode once a period of time report state of all its block. Again, unless cluster is not too big it should not be a problem.
0 0
- several technical limits to the NameNode
- HDFS scalability:the limits to growth
- Introduction to the t Distribution (non-technical)
- HDFS scalability: the limits to growth 学习笔记
- Defeating the Hacker : A Non-Technical Guide to Computer Security
- Defeating the Hacker : A non-technical guide to computer security
- Why Blogging --- the reason to write technical blogs
- The Limits of Agile
- Surviving the Technical Interview
- several ways to reduce the amount of bandwidth consumed by an app
- the several ways of Base64
- the several Japan scorching springs
- The J2EE Architect's Handbook: How to be a Successful Technical Architect for J2EE Applications
- C# Need to understand some of the technical (C# 需要懂的一些技术)
- how to express technical issues
- How to write technical articles
- From Developer to Technical Manager
- several ways to solve np-complete problems
- MyEclipse 修改注释模版
- webservice文件上传下载(byte[] 实现方式)
- 心是净土,也是红尘
- maven在对resources目录下的文件进行过滤时出现部分文件未能正确替换
- CSS A link hover active visited伪类超链接锚文本样式教程
- several technical limits to the NameNode
- windows phone 8 开发学习笔记(三)
- android developer镜像网站
- 实战总结经验分享,同磁盘下安装多linux添加启动项到grub
- likely(x)
- Android的Listener如何应用。
- 途径岁月,与光阴说禅
- C++命名规则
- 嵌套的异步回调