hadoop ganglia configuration

作者：hovlj_1130 | 可以任意转载, 但转载时务必以超链接形式标明文章原始出处 和 作者信息 及 版权声明
http://hi.baidu.com/hovlj_1130/blog/item/e8fe89c3e9a67e160ff47755.html

#准备工作
场景：
给hadoop集群机器安装监控工具ganglia，采集cpu、memory、disk、process、network数据的同时，采集hadoop相关的数值(metrics)
server:hadoopbj01~hadoopbj08
hadoop-version:hadoop-0.20.2
ganglia-version:ganglia-3.1.1

准备工作:
在ganglia服务端需要安装gmetad的机器(下文称之为ganglia_m)上rhn_register(需要redhat注册码)，以便使用yum install，要不会被依赖包折腾的死无活来的。

step1:(in ganglia_m)
#安装基础依赖包
yum -y install apr-devel apr-util check-devel cairo-devel pango-devel libxml2-devel \
rpmbuild glib2-devel dbus-devel freetype-devel fontconfig-devel gcc-c++ expat-devel \
python-devel libXrender-devel pcre pcre-devel

step2:(in ganglia_m)
yum install libconfuse libconfuse-devel

step3:(in ganglia_m)
#安装RRDTool

wget http://dag.wieers.com/rpm/packages/rpmforge-release/rpmforge-release-0.3.6-1.el5.rf.i386.rpm

rpm -ivh rpmforge-release-0.3.6-1.el5.rf.i386.rpm

会在/etc/yum.repos.d目录生成一些yum源

yum install rrdtool

yum install rrdtool-devel

which rrdtool
ldconfig  -p | grep rrd # make sure you have the new rrdtool libraries linked.

step4:(in ganglia_m: 包含 步骤0~4)
#步骤 0：安装ganglia
cd /tmp/
wget http://sourceforge.net/projects/ganglia/files/ganglia%20monitoring%20core/3.1.1%20\(Wien\)/ganglia-3.1.1.tar.gz/download
cd /tmp/
tar zxvf ganglia*gz
cd ganglia-3.1.1/
./configure --with-gmetad
make -j8
make install

#步骤 1：处理命令行文件
cd /tmp/ganglia-3.1.1/   # you should already be in this directory
mkdir -p /var/www/html/ganglia/  # make sure you have apache installed
cp -a web/* /var/www/html/ganglia/   # this is the web interface
cp gmetad/gmetad.init /etc/rc.d/init.d/gmetad  # startup script
cp gmond/gmond.init /etc/rc.d/init.d/gmond
mkdir /etc/ganglia  # where config files go
gmond -t | tee /etc/ganglia/gmond.conf  # generate initial gmond config
cp gmetad/gmetad.conf /etc/ganglia/  # initial gmetad configuration
mkdir -p /var/lib/ganglia/rrds  # place where RRDTool graphs will be stored
chown nobody:nobody /var/lib/ganglia/rrds  # make sure RRDTool can write here.
chkconfig --add gmetad  # make sure gmetad starts up at boot time
chkconfig --add gmond # make sure gmond starts up at boot time

#步骤 2：修改 /etc/ganglia下的gmond.conf和gmetad.conf
vi gmetad.conf
#添加需要监控的机器，以及监控组名，下面的列，表示建立监控组：hadoop-bj，监控机器hadoopbj01~hadoopbj08
data_source "hadoop-bj" hadoopbj01 hadoopbj02 hadoopbj03 hadoopbj04 hadoopbj05 hadoopbj06 hadoopbj07 hadoopbj08

vi gmond.conf
#现在可以修改 /etc/ganglia/gmond.conf 以命名集群。我们上面定义的集群名称为"hadoop-bj"，因此我们将name="unspecified"更改为name="hadoop-bj"
#这里我们的udp_send_channel和udp_recv_channel都是用默认的mcast_join和Port，当然你可以修改
cluster {
name = "hadoop-bj"
owner = "unspecified"
latlong = "unspecified"
url = "unspecified"
}

udp_send_channel {
mcast_join = 239.2.11.71
port = 8649
ttl = 1
}

udp_recv_channel {
mcast_join = 239.2.11.71
port = 8649
bind = 239.2.11.71
}

#步骤 3：注意多宿主计算机
#在我的集群中，eth0 是我的系统的公共 IP 地址。但是，监视服务器将通过 eth1 与私有集群网络中的节点进行通信。我需要确保 Ganglia 使用的多点传送将与 eth1 绑定在一起。这可以通过创建 /etc/sysconfig/network-scripts/route-eth1 文件来完成。添加 239.2.11.71 dev eth1 内容。
#然后您可以使用 service network restart 重新启动网络并确保路由器显示此 IP 通过 eth1。注：您应当使用 239.2.11.71，因为这是 ganglia 的默认多点传送通道。如果使用其他通道或者增加更多通道，请更改它。
touch /etc/sysconfig/network-scripts/route-eth1
echo "239.2.11.71 dev eth1">>/etc/sysconfig/network-scripts/route-eth1
service network restart

#步骤 4：在管理服务器中启动它
service gmond start
service gmetad start
service httpd restart

step5:(in ganglia_m)
#通过脚本修改配置需要监控的节点，注意我们这里ganglia_m同时也为hadoopbj01，已经配置好gmond，因此下面的脚本中只需配置hadoopbj02~hadoopbj08
touch /tmp/mynodes
vi  /tmp/mynodes
hadoopbj02
hadoopbj03
hadoopbj04
hadoopbj05
hadoopbj06
hadoopbj07
hadoopbj08

touch /tmp/configure-all-ganglia
vi /tmp/configure-all-ganglia
for i in `cat /tmp/mynodes`; do 
scp /usr/sbin/gmond $i:/usr/sbin/gmond
ssh $i "mkdir -p /etc/ganglia/"
scp /etc/ganglia/gmond.conf $i:/etc/ganglia/
scp /etc/init.d/gmond $i:/etc/init.d/
scp /usr/lib64/libganglia-3.1.1.so.0 $i:/usr/lib64/
scp /lib64/libexpat.so.0 $i:/lib64/
scp /usr/lib64/libconfuse.so.0 $i:/usr/lib64/
scp /usr/lib64/libapr-1.so.0 $i:/usr/lib64/
scp -r /usr/lib64/ganglia $i:/usr/lib64/
scp /etc/sysconfig/network-scripts/route-eth1 $i:/etc/sysconfig/network-scripts/
ssh $i "service network restart"
ssh $i "service gmond start"
ssh $i "chkconfig --add gmond"
done
sh /tmp/configure-all-ganglia

#查看运行情况
http://hadoopbj01/ganglia/

step6:(in ganglia_m)
#我们这里使用的是hadoop-0.20.2，在使用ganglia-3.1.1收集hadoop数据时需要打补丁HADOOP-4756
#下面我们给hadoop打补丁
#打补丁的事情，还有下面的重新ant package最好在测试环境先测试通过，然后再将生成的hadoop core包复制到在线环境替换即可。
wget https://issues.apache.org/jira/browse/HADOOP-4756
cp HADOOP-4675-v9.patch HADOOP-4675-v9.patch.bak
vi HADOOP-4675-v9.patch
#下载下来的补丁需要修改一下，下面是修改的细节，修改后编译生成新的hadoop core包，里面包含最新的org.apache.hadoop.metrics.ganglia.GangliaContext31类
#用org.apache.hadoop.metrics.ganglia.GangliaContext包装的；
#（为什么下载下来的补丁不能直接patch，大家是什么情况？因为这个补丁被折腾了好久。）
diff HADOOP-4675-v9.patch HADOOP-4675-v9.patch.bak 
237c237
< Index: src/core/org/apache/hadoop/metrics/ganglia/GangliaContext.java
---
> Index: src/java/org/apache/hadoop/metrics/ganglia/GangliaContext.java
239,240c239,240
< --- src/core/org/apache/hadoop/metrics/ganglia/GangliaContext.java    (revision 771522)
< +++ src/core/org/apache/hadoop/metrics/ganglia/GangliaContext.java    (working copy)
---
> --- src/java/org/apache/hadoop/metrics/ganglia/GangliaContext.java    (revision 771522)
> +++ src/java/org/apache/hadoop/metrics/ganglia/GangliaContext.java    (working copy)
325c325
< Index: src/core/org/apache/hadoop/metrics/ganglia/GangliaContext31.java
---
> Index: src/java/org/apache/hadoop/metrics/ganglia/GangliaContext31.java
327,328c327,328
< --- src/core/org/apache/hadoop/metrics/ganglia/GangliaContext31.java    (revision 0)
< +++ src/core/org/apache/hadoop/metrics/ganglia/GangliaContext31.java    (revision 0)
---
> --- src/java/org/apache/hadoop/metrics/ganglia/GangliaContext31.java    (revision 0)
> +++ src/java/org/apache/hadoop/metrics/ganglia/GangliaContext31.java    (revision 0)
cd $HADOOP_HOME
patch -p0 < HADOOP-4675-v9.patch
vi build.xml
#注释掉904和908行
#（ant不熟，同事说904,908编译通不过没关系，为什么？最好自己写一个build.xml，生成包含org.apache.hadoop.metrics.ganglia.GangliaContext31类的hadoop core包，我不会写...只好用自带的。）
<target name="forrest.check" unless="forrest.home" depends="java5.check">
<!--fail message="'forrest.home' is not defined. Please pass -Dforrest.home=&lt;base of Apache Forrest installation&gt; to Ant on the command-line." /-->
</target>

<target name="java5.check" unless="java5.home">
<!--fail message="'java5.home' is not defined.  Forrest requires Java 5.  Please pass -Djava5.home=&lt;base of Java 5 distribution&gt; to Ant on the command-line." /-->
</target>
ant package
#ant package成功后，会生成build目录，下面有编译生成的新的hadoop core包
#将新生成的带有org.apache.hadoop.metrics.ganglia.GangliaContext31类的hadoop core jar包替换原有的包
cp $HADOOP_HOME/hadoop-0.20.2-core.jar $HADOOP_HOME/hadoop-0.20.2-core.jar.bak
cp $HADOOP_HOME/build/hadoop-0.20.3-dev-core.jar $HADOOP_HOME/

#更新hadoop-metrics.properties,使用新的GangliaContext31类收集hadoop数据
vi $HADOOP_HOME/conf/hadoop-metrics.properties
# Configuration of the "dfs" context for ganglia
dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
dfs.period=10
dfs.servers=239.2.11.71:8649

# Configuration of the "mapred" context for null
#mapred.class=org.apache.hadoop.metrics.spi.NullContext

# Configuration of the "mapred" context for file
#mapred.class=org.apache.hadoop.metrics.file.FileContext
#mapred.period=10
#mapred.fileName=/tmp/mrmetrics.log

# Configuration of the "mapred" context for ganglia
mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
mapred.period=10
mapred.servers=239.2.11.71:8649


# Configuration of the "jvm" context for null
#jvm.class=org.apache.hadoop.metrics.spi.NullContext

# Configuration of the "jvm" context for file
#jvm.class=org.apache.hadoop.metrics.file.FileContext
#jvm.period=10
#jvm.fileName=/tmp/jvmmetrics.log

# Configuration of the "jvm" context for ganglia
jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
jvm.period=10
jvm.servers=239.2.11.71:8649

rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
rpc.period=10
rpc.servers=239.2.11.71:8649

#重启hadoop集群
#namenode
$HADOOP_HOME/bin/stop-dfs.sh
#jobtracker
$HADOOP_HOME/bin/stop-mapred.sh
#namenode
$HADOOP_HOME/bin/start-dfs.sh
#jobtracker
$HADOOP_HOME/bin/start-mapred.sh
#重启ganglia集群
service gmond restart
service gmetad restart
service httpd restart

#不出意外的话，这时候ganglia可以收集hadoop metrics了。

参考url:
#http://wiki.yepn.net/ganglia#%E9%85%8D%E7%BD%AE_node%E7%9A%84hadoop%E7%9B%91%E6%8E%A7
#http://www.pginjp.org/modules/newbb/viewtopic.php?topic_id=1235&forum=22
#http://www.pginjp.org/modules/newbb/viewtopic.php?topic_id=1234&forum=7
#http://www.ibm.com/developerworks/cn/linux/l-ganglia-nagios-1/index.html
#sed用法 http://hi.baidu.com/hovlj_1130/blog/item/e9721b7b31e8dbe02e73b3b2.html