hadoop基准测试
来源:互联网 发布:房屋租赁哪个软件 编辑:程序博客网 时间:2024/04/29 02:19
写测试hadoop jarhadoop-0.20.2-test.jar TestDFSIO -write -nrFiles 10 -fileSize 1000
----- TestDFSIO ----- : write
Date & time: Fri Jul 24 14:24:36 CST 2015
Number of files: 10
Total MBytes processed: 10000
Throughput mb/sec: 24.302163378583963
Average IO rate mb/sec: 24.46268653869629
IOrate std deviation: 2.0279575539782315
Test exec time sec: 72.853
读测试hadoop jarhadoop-0.20.2-test.jar TestDFSIO -read -nrFiles 10 -fileSize 1000
----- TestDFSIO ----- : read
Date & time: Fri Jul 24 14:27:11 CST 2015
Number of files: 10
Total MBytes processed: 10000
Throughput mb/sec: 80.73566336457803
Average IO rate mb/sec: 106.7965087890625
IOrate std deviation: 70.18198197030486
Test exec time sec: 46.772
清除数据
[root@ZhejiangYiwuF1210 hadoop]# hadoop jarhadoop-0.20.2-test.jar TestDFSIO -clean
排序测试
1.在各节点上分别运行2个map任务,每个节点产生1GB大小的随机二进制数据,并输出到目录/examples/terasort-input
hadoop jar hadoop-0.20.2-examples.jar teragen 10000000/examples/terasort-input
Launched map tasks=2
FileSystemCounters
HDFS_BYTES_WRITTEN=1000000000
Map-Reduce Framework
Map input records=10000000
Spilled Records=0
Map input bytes=10000000
Map output records=10000000
MapReduce(map:2个,reduce:0个)
2.排序,并将结果输出到目录/examples/terasort-output
#hadoop jar hadoop-0.20.2-examples.jar terasort /examples/terasort-input/examples/terasort-output
MapReduce(map:16个,reduce:1个)
3.检查是否已经排好序
# hadoop jar hadoop-0.20.2-examples.jarteravalidate /examples/terasort-output /examples/terasort-validate
MapReduce(map:1个,reduce:1个)
Namenode的负载测试
# hadoop jar hadoop-0.20.2-test.jarnnbench -operation create_write -maps 12 -reduces 6 -blockSize 1 -bytesToWrite0 -numberOfFiles 1000 -replicationFactorPerFile 3 -readFileAfterOpen true-baseDir /benchmarks/NNBench-`hostname -s`
-------------- NNBench -------------- :
Version:NameNode Benchmark 0.4
Date& time: 2015-07-24 14:52:04,989
TestOperation: create_write
Starttime: 2015-07-24 14:48:46,912
Maps torun: 12
Reducesto run: 6
BlockSize (bytes): 1
Bytes towrite: 0
Bytesper checksum: 1
Numberof files: 1000
Replicationfactor: 3
Successfulfile operations: 0
MapReduce(map:12个,reduce:6个)
MapReduce连续性测试
mrbench会多次重复执行一个小作业,用于检查在机群上小作业的运行是否可重复以及运行是否高效
运行一个小作业10次
hadoop jar hadoop-0.20.2-test.jar mrbench -numRuns 10
DataLines Maps Reduces AvgTime (milliseconds)
1 2 1 29637
以上结果表示平均作业完成时间是29637(milliseconds)
文件系统一致性的分布式检查
hadoop jarhadoop-0.20.2-test.jar DistributedFSCheck
----- DistributedFSCheck ----- :
Date & time: Fri Jul 2415:04:18 CST 2015
Total number of blocks: 58
Total number of files: 3171
Number of corrupted blocks: 0
Number of corrupted files: 0
MapReduce(map:2个,reduce:1个)
- hadoop基准测试
- Hadoop Gridmix基准测试
- 运行hadoop基准测试
- hadoop基准测试---新手入门
- 运行hadoop基准测试
- hadoop基准测试
- Hadoop 集群基准测试
- hadoop基准测试
- hadoop基准性能测试
- 运行hadoop基准测试
- Hadoop基准测试
- Hadoop基准测试
- hadoop基准测试
- 运行基准测试hadoop
- Hadoop基准测试工具
- Hadoop基准测试
- Hadoop基准测试
- hadoop集群基准测试
- 土木工程转行IOS的点点滴滴,没受伤是不知道回头滴
- ANR相关的所有知识 这个才是我想要的
- Number 格式化 ,精确加减
- ORA-01652: unable to extend temp segment by 128 in tablespace TB1
- 2. 重建二叉树
- hadoop基准测试
- java中如何从字符串中删选中文字符(GBK编码和UTF-8编码)
- jsoncpp-0.5,支持uint64
- 使用BeautifulSoup实现简单豆瓣爬虫
- java中Cookie的使用
- Share folder between windows and ubuntu linux vm from Virtual Box
- 2012年5月SAT香港真题解析
- 土木工程转行IOS的点点滴滴,没受伤是不知道回头滴
- AngularJS 的表单验证