mapreduce counter
来源:互联网 发布:淘宝联盟刷引流人数 编辑:程序博客网 时间:2024/05/17 06:57
http://diveintodata.org/2011/03/15/an-example-of-hadoop-mapreduce-counter/
http://stackoverflow.com/questions/8616041/how-do-i-get-counters-in-hadoop
http://blog.csdn.net/posa88/article/details/7904720
mark
If you are submitting your job like this:
Configuration conf = new Configuration(); Job job = new Job(conf); job.waitForCompletion(true);
And it has finished (you can call this even when its running, but the results won't be final then, because the job hasn't completed yet.), you can grab the counters by using:
long counter = job.getCounters().findCounter(ExplorationReducer.UpdateCounter.UPDATED) .getValue();
This is the name of the enum counter I used in my job:
ExplorationReducer.UpdateCounter.UPDATED
If you want all counter you have to traverse the backing structure behind the Counters
object. There is an iterator for it.
An Example of Hadoop MapReduce Counter
Posted: March 15, 2011 Filed under:FOSS| Tags: counter, hadoop, mapreduce 2 CommentsMapReduce Counter
Hadoop MapReduce Counter provides a way to measure the progress or the number of operations that occur within MapReduce programs. Basically, MapReduce framework provides a number of built-in counters to measure basic I/O operations, such as FILE_BYTES_READ/WRITTEN and Map/Combine/Reduce input/output records. These counters are very useful especially when you evaluate some MapReduce programs. Besides, the MapReduce Counter allows users to employ your own counters. Since MapReduce Counters are automatically aggregated over Map and Reduce phases, it is one of the easiest way to investigate internal behaviors of MapReduce programs. In this post, I’m going to introduce how to use your own MapReduce Counter. The example sources described in this post are based on Hadoop 0.21 API.
Incrementing your counter
For your own MapReduce counter, you first define a enum type as follow:
public
static
enum
MATCH_COUNTER {
INCOMING_GRAPHS,
PRUNING_BY_NCV,
PRUNING_BY_COUNT,
PRUNING_BY_ISO,
ISOMORPHIC
};
And then, when you want to increment your own counter, you should call the incrementmethod as follows:
context.getCounter(MATCH_COUNTER.INCOMING_GRAPHS).increment(
1
);
You can access context instance within setup, cleanup,map, andreduce method in Mapper or Reducer class. You can get a desired counter via calling context.getCounter method with some enum value.
Finding your counter
You can get some Counters from a finished job as follows:
Configuration conf =
new
Configuration();
Cluster cluster =
new
Cluster(conf);
Job job = Job.getInstance(cluster,conf);
result = job.waitForCompletion(
true
);
...
Counters counters = job.getCounters();
The instance of Counters class contains all of the counters obtained from a job. So, when you want to get your own counter, you should callfindCounter method with aenum type as follows:
Counter c1 = counters.findCounter(MATCH_COUNTER.INCOMING_GRAPHS);
System.out.println(c1.getDisplayName()+
":"
+c1.getValue());
The below example shows how to get built-in counter groups that Hadoop provides basically.
for
(CounterGroup group : counters) {
System.out.println(
"* Counter Group: "
+ group.getDisplayName() +
" ("
+ group.getName() +
")"
);
System.out.println(
" number of counters in this group: "
+ group.size());
for
(Counter counter : group) {
System.out.println(
" - "
+ counter.getDisplayName() +
": "
+ counter.getName() +
": "
+counter.getValue());
}
}
- mapreduce counter
- MapReduce中的counter
- MapReduce-Counter使用
- MapReduce:默认Counter的含义
- MapReduce:默认Counter的含义
- MapReduce:默认Counter的含义
- MapReduce:默认Counter的含义
- MapReduce:默认Counter的含义
- MapReduce:默认Counter的含义
- MapReduce默认Counter的含义
- MapReduce:默认Counter的含义
- An Example of Hadoop MapReduce Counter
- Hadoop实践(三)---MapReduce中的Counter
- Hadoop中MapReduce中的counter的意义
- counter
- counter
- Counter
- MapReduce Counter监控窗口各个参数的含义
- linux nginx开机启动
- mysql中group by的时候字段不加聚合函数和distinct的情况
- _raw_readl() 与ioread32()有什么区别
- android SQLite使用SQLiteOpenHelper类对数据库进行操作
- 开源游戏 “Elvish Bird”
- mapreduce counter
- STL中各个容器的内存的释放
- 如何才能不纠结
- java中的向下转型和向上转型
- 给Win32 GUI程序增加控制台窗口的方法
- run configurations改端口
- LoadRunner参数化详解
- spi流程
- 关于正则表达式的个人笔记