several technical limits to the NameNode

来源:互联网 发布:java工控软件开发 编辑:程序博客网 时间:2024/06/03 15:58
There are several technical limits to the NameNode (NN), and facing any of them will limit your scalability.
1. Memory. NN consume about 150 bytes per each block. From here you can calculate how much RAM you need for your data. There is good discussion: Namenode file quantity limit
2. IO. NN is doing 1 IO for each change to filesystem (like create, delete block etc). So your local IO should allow enough. It is harder to estimate how much you need. Taking into account fact that we are limited in number of blocks by memory you will not claim this limit unless your cluster is very big. If it is - consider SSD.
3. CPU. Namenode has considerable load keeping track of health of all blocks on all datanodes. Each datanode once a period of time report state of all its block. Again, unless cluster is not too big it should not be a problem.
0 0
原创粉丝点击