hadoop集群配置机架感知

来源：互联网发布：数据集的稀疏性编辑：程序博客网时间：2024/04/27 15:03

自己搭建的hadoop集群一般默认不具备机架感知能力，为了提高hadoop集群的网络性能，我们常常需要配置机架感知。下面是以下步骤：

一、修改配置文件core-site.xml，添加配置属性：

<property><name>net.topology.script.file.name</name><value>/home/hadoop-2.4.1/rackaware.py</value></property>

二、编写自己的脚本rackaware.py：

#!/usr/bin/python#-*-coding:UTF-8 -*-import sysrack = {        "14a":"rack1","14b":"rack1","14c":"rack1","14d":"rack1","14e":"rack1","14f":"rack1","14g":"rack1","14h":"rack1",        "14i":"rack1","14j":"rack1","14k":"rack1","14l":"rack1","14m":"rack1","14n":"rack1","14o":"rack1","14p":"rack1",        "14q":"rack1","14r":"rack1","14s":"rack1","14t":"rack1","14u":"rack1","14v":"rack1","14w":"rack1","14x":"rack1",        "15a":"rack2","15b":"rack2","15c":"rack2","15d":"rack2","15e":"rack2","15f":"rack2","15g":"rack2","15h":"rack2",        "15i":"rack2","15j":"rack2","15k":"rack2","15l":"rack2","15m":"rack2","15n":"rack2","15o":"rack2","15p":"rack2",        "15q":"rack2","15r":"rack2","15s":"rack2","15t":"rack2","15u":"rack2","15v":"rack2","15w":"rack2","15x":"rack2",        "16a":"rack3","16b":"rack3","16c":"rack3","16d":"rack3","16e":"rack3","16f":"rack3","16g":"rack3","16h":"rack3",        "16i":"rack3","16j":"rack3","16k":"rack3","16l":"rack3","16m":"rack3","16n":"rack3","16o":"rack3","16p":"rack3",        "16q":"rack3","16r":"rack3","16s":"rack3","16t":"rack3","16u":"rack3","16v":"rack3","16w":"rack3","16x":"rack3",       }if __name__=="__main__":    print "/" + rack.get(sys.argv[1],"rack0")

注：脚本中可以写主机名也可以写ip地址。

三、重启namenode。

转载自http://www.csdn123.com/html/itweb/20130826/81942_81949_81957.htm

0 0