《Hadoop The Definitive Guide》ch10 Administering Hadoop

1. 集群基准测试程序

c01s02h01ate1:nomad2 # hadoop jar /local/hadoop/hadoop-0.20.203.0/hadoop-test-0.20.203.0.jar An example program must be given as the first argument.Valid program names are:  DFSCIOTest: Distributed i/o benchmark of libhdfs.  DistributedFSCheck: Distributed checkup of the file system consistency.  MRReliabilityTest: A program that tests the reliability of the MR framework by injecting faults/failures  TestDFSIO: Distributed i/o benchmark.  dfsthroughput: measure hdfs throughput  filebench: Benchmark SequenceFile(Input|Output)Format (block,record compressed and uncompressed), Text(Input|Output)Format (compressed and uncompressed)  loadgen: Generic map/reduce load generator  mapredtest: A map/reduce test check.  mrbench: A map/reduce benchmark that can create many small jobs  nnbench: A benchmark that stresses the namenode.  testarrayfile: A test for flat files of binary key/value pairs.  testbigmapoutput: A map/reduce program that works on a very big non-splittable file and does identity map/reduce  testfilesystem: A test for FileSystem read/write.  testipc: A test for ipc.  testmapredsort: A map/reduce program that validates the map-reduce framework's sort.  testrpc: A test for rpc.  testsequencefile: A test for flat files of binary key value pairs.  testsequencefileinputformat: A test for sequence file input format.  testsetfile: A test for flat files of binary key/value pairs.  testtextinputformat: A test for text input format.  threadedmapbench: A map/reduce benchmark that compares the performance of maps with multiple spills over maps with 1 spill

c01s02h01ate1:nomad2 # hadoop jar /local/hadoop/hadoop-0.20.203.0/hadoop-test-0.20.203.0.jar DFSCIOTestTestDFSIO.0.0.4Usage: TestDFSIO -read | -write | -clean [-nrFiles N] [-fileSize MB] [-resFile resultFileName] [-bufferSize Bytes] 

2.  safe mode

c01s02h01ate1:nomad2 # hadoop dfsadmin -safemode getSafe mode is OFF

c01s02h01ate1:nomad2 # hadoop dfsadmin -refreshNodes

3.  web界面

http://c01s0201ate2:50075/blockScannerReport

http://c01s0201ate1:50030/logLevel?log=org.apache.hadoop.mapred.JobTracker

http://c01s0201ate1:50030/stacks

4. 负载平衡

c01s02h01ate1:nomad2 # start-balancer.sh starting balancer, logging to /local/hadoop/hadoop-0.20.203.0/bin/../logs/hadoop-nomad2-balancer-c01s02h01ate1.outTime Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being MovedThe cluster is balanced. Exiting...Balancing took 413.0 milliseconds