HDPCD-Java-复习笔记（15）

Unit Testing

MRUnit is an Apache project that provides Java classes for unit testing MapReduce jobs.

There are basically three types of tests that can be run with MRUnit:

Map test

Test a Mapper class.

Reduce test

Test a Reducer class.

MapReduce

Send a record to a Mapper and verify the Reducer’s output.

The MRUnit classes are found in the org.apache.hadoop.mrunit package. The three types of unit tests areeach implemented by the driver classes found in the org.apache.hadoop.mrunit.mapreduce package:

•MapDriver
• ReduceDriver
• MapReduceDriver

To setup and run an MRUnit test, there are four essential steps:

1.Create an instance of the driver class, passing in an instance of the Mapper class, Reducer class, or, for a MapReduce test, both.

2.Use the addInput or withInput methods of the driver class to specify the < key ,value > pair to pass into the map or reduce method.

3.Use the addOutput or withOutput methods in the driver class to specify the expected output.

4.Invoke the run method of the driver to run the test.

Using MRUnit with JUnit

To use MRUnit with JUnit, add the junit and org.apache.mrunit frameworks to your build environment. 

For example, if you are using Gradle the settings look like:

• testCompile group: 'junit', name: 'junit',  
•   version:'4.11'
•  
• testCompile group: 'org.apache.mrunit', name:
•   'mrunit', version:'1.0.0', classifier:'hadoop2'

Setting Up a Test

A unit test class typically stores the driver instance as a field and initializes it in a @Before method.

For example, the following EmployeeJobTest class creates a ReduceDriver instance:

• public class EmployeeJobTest {
• ReduceDriver<EmployeeKey, Employee,
•               Text, DoubleWritable> reduceDriver;
•     @Before
•     public void init() {
•         PayrollReducer reducer = new PayrollReducer();
•         reduceDriver = ReduceDriver.newReduceDriver(reducer);
•     }
•     //remainder of class definition...
• }

Testing a Mapper

The example above uses the runTest method, which validates the input with the expected output. The test can also be run using the run method, which returns the results in a List<Pair> instance instead of validating the results.

• public class EmployeeJobTest {
• MapDriver<LongWritable, Text,
•            EmployeeKey, Employee> mapDriver;
•     @Before
•     public void init() {
•         mapDriver = MapDriver.newMapDriver(
•          new PayrollMapper());
•     }
•     @Test
•     public void testPayrollMapperInput() {
•          LongWritable inputKey = new LongWritable(0);
•         Text inputValue =
•             new Text("111223333,John,Wayne,IT,50000.00");
•         EmployeeKey outputKey = new EmployeeKey(111223333);
•         Employee outputValue =
•             new Employee("John","Wayne","IT",50000.00);
•         mapDriver.withInput(inputKey, inputValue);
•         mapDriver.withOutput(outputKey, outputValue);
•         mapDriver.runTest();
•     }
• }

Testing a Reducer

•public class EmployeeJobTest {
• ReduceDriver<EmployeeKey, Employee,
•     Text, DoubleWritable> reduceDriver;
•     @Before
•     public void init() {
•         PayrollReducer reducer = new PayrollReducer();
•         reduceDriver = ReduceDriver.newReduceDriver(reducer);
•     }
•     @Test
•     public void testPayrollReducer() throws IOException {
•         List<Employee> values = new ArrayList<Employee>();
•         values.add(mapOutputValue);
•         reduceDriver.withInput(mapOutputKey, values);
•         List<Pair<Text,DoubleWritable>> result =
•             reduceDriver.run();
•         for(Pair<Text,DoubleWritable> pair :result) {
•         System.out.println(pair.getFirst().toString()
•              + " " + pair.getSecond().toString());
•        }
•     }

•}

A MapReduce Test

• public class EmployeeJobTest {
•     MapReduceDriver<LongWritable, Text, EmployeeKey,
•                  Employee, Text, DoubleWritable> mrDriver;
•      LongWritable inputKey = new LongWritable(0);
•     Text inputValue =
•         new Text("111223333,John,Wayne,IT,50000.00");
•     EmployeeKey mapOutputKey = new EmployeeKey(111223333);
•     Employee mapOutputValue =
•        new Employee("John","Wayne","IT",50000.00);
•     @Before
•      public void init() {
•          PayrollReducer reducer = new PayrollReducer();
•          PayrollMapper mapper = new PayrollMapper();
•         mrDriver = MapReduceDriver.newMapReduceDriver(mapper, reducer);
•     }
•     @Test
•     public void testPayrollJob() {
•         mrDriver.withInput(inputKey, inputValue);
•         Text outputKey = new Text("Wayne,John");
•          DoubleWritable outputValue =
•             new DoubleWritable(50000.00);
•         mrDriver.withOutput(outputKey, outputValue);
•         mrDriver.runTest();
•     }
• }