Hadoop集成BTrace
来源:互联网 发布:程序员喜欢用的浏览器 编辑:程序博客网 时间:2024/05/21 07:47
1.BTrace
1.1BTrace部署
1.下载BTrace工具包,官网地址:https://kenai.com/projects/btrace/downloads/directory/releases
2.解压btrace-bin.tar.gz,命令如下:
$ tar -zxvf btrace-bin.tar.gz
3.配置环境变量
BTRACE_HOME=/usr/btrace-binPATH=$PATH:$BTRACE_HOME/bin
4.执行btrace(注意:$BTRACE_HOME/bin下btrace、btracec、btracer文件权限)命令如下,表示部署成功:
1.2BTrace脚本
BTrace测试脚本代码:
importcom.sun.btrace.annotations.*; import static com.sun.btrace.BTraceUtils.*; @BTracepublic class TraceMR{ @TLS privatestaticlongmapperSetupStartTime = 0l; @OnMethod(clazz="org.apache.hadoop.mapreduce.Mapper", method="run", location=@Location(where=Where.BEFORE, value=Kind.CALL,clazz="/.*/", method="setup")) public static void onMapper_run_Before_Call_setup() { mapperSetupStartTime =timeNanos(); println(strcat("MAP\tSTARTUP_MEM\t",str(used(heapUsage())))); } @OnMethod(clazz="org.apache.hadoop.mapreduce.Mapper", method="run", location=@Location(where=Where.AFTER, value=Kind.CALL,clazz="/.*/", method="setup")) public static void onMapper_run_After_Call_setup() { println(strcat("MAP\tSETUP\t",str(timeNanos() -mapperSetupStartTime))); println(strcat("MAP\tSETUP_MEM\t",str(used(heapUsage())))); }}
1.3编译BTrace脚本
编译命令:
$ btracec TraceMR.java
2.Hadoop配置
2.1分发文件
将BTrace文件:$BTRACE_HOME/build目录下的btrace-agent.jar和btrace-boot.jar,以及编译Btrace脚本后生成的class文件分发到各个子节点上。建议使用脚本分发,代码如下:
#!/usr/bin/envbash#Usage:# ./install_btrace.sh <slaves_file># # where:# slaves_file = File containing a list of slavemachines##Example:# ./install_btrace.sh /root/SLAVE_NAMES.txt#Make sure we have all the argumentsif [$# -ne 1 ]; then echo "Usage: $0<slaves_file>" echo " slaves_file = File containing a list of slave machines" echo "" exit -1fi #Get the slaves filedeclareSLAVES_FILE=$1;iftest ! -e $SLAVES_FILE; then echo "ERROR: The file '$SLAVES_FILE'does not exist. Exiting" exit -1fi#Get the slaves filedeclareSLAVES_FILE=$1;iftest ! -e $SLAVES_FILE; then echo "ERROR: The file '$SLAVES_FILE'does not exist. Exiting" exit -1fi MASTER_BTRACE_DIR=/home/wangpeng/zhiming/usr/btrace/buildSLAVES_BTRACE_DIR=/home/wangpeng/zhiming/usr forslave in `cat "$SLAVES_FILE"`; do{ printf "Installing on host:$slave\n" ssh $slave "mkdir -p$SLAVES_BTRACE_DIR" scp ${MASTER_BTRACE_DIR}/btrace-agent.jar$slave:$SLAVES_BTRACE_DIR/. scp ${MASTER_BTRACE_DIR}/btrace-boot.jar$slave:$SLAVES_BTRACE_DIR/. scp ${MASTER_BTRACE_DIR}/TraceMR.class$slave:$SLAVES_BTRACE_DIR/.}done
2.2配置Hadoop文件
在所有子节点的$HADOOP_HOME安装目录下找到mapred-site.xml文件,进行如下设置:
代码:
<property> <name>mapred.map.child.java.opts</name> <value> -Xmx1024m -javaagent:/home/wangpeng/zhiming/usr/btrace/build/btrace-agent.jar=script=/home/wangpeng/zhiming/usr//TraceMR.class,stdout=true,noServer=true </value></property>
2.3运行MR程序
- 准备一个写好的MR程序,如WordCount.java程序;
- .执行hadoop jar WordCount.jar WordCount /user/wzm/input /user/wzm/output。
2.4查看输出文件
在所有子节点的$HADOOP_HOME安装目录下找到logs目录中的stdout文件,执行命令:cat stdout,且有如下输出表示配置成功:
1 0
- Hadoop集成BTrace
- BTrace
- Btrace
- BTrace
- btrace
- btrace
- btrace
- 集成Hadoop和Hypertable
- eclipse集成hadoop
- Ambari集成Hadoop安装
- hadoop eclipse集成开发
- Hadoop集成Hue详解
- Hadoop集群集成kerberos
- 集成Hadoop到数据中心
- hadoop集成RHive
- kerberos 与Hadoop集成
- eclipse集成Hadoop开发
- Hadoop集群集成kerberos
- 算法的封装与切换——策略模式(四)
- UVA 10036 Divisibility
- 小白学开发(iOS)基础C_字符串 (2015-07-16)
- Node.JS的异步编程风格
- swf 文件解析
- Hadoop集成BTrace
- 数的排列的递归实现-C++ STL实现
- 为什么你不需要做一名全栈工程师?
- 第三天 循环语句
- myeclipse中手工添加dtd支持
- 开通我的博客,记录一下
- 自己总结的web前端知识体系大全【欢迎补充】
- centOS 7 network sshd 服务
- 利用阿里云提供的镜像快速更换本地的yum源