Using Multi-Threaded Tests

http://groboutils.sourceforge.net/testing-junit/using_mtt.html

The GroboUtils class MultiThreadedTestRunner was based on thearticle "JUnit Best Practices" by Andy Schneider([e-mail]), published online athttp://www.javaworld.com/javaworld/jw-12-2000/jw-1221-junit_p.html.Since GroboUtils first started using that implementation, many changes haveoccured in the code to make a more robust and stable testing environment.Due to these changes, the use of the class will be fully described in thisdocument.

Let's start by testing a sample application:

   1:import java.io.*;   2:public class WriterCache   3:{   4:    public static interface IWriterFactory   5:    {   6:        public Writer createWriter( String location )   7:            throws IOException;   8:    }   9:      10:    private String openLocation;  11:    private Writer w;  12:    private IWriterFactory factory;  13:    private volatile int openCount = 0;  14:      15:    public WriterCache( IWriterFactory wf )  16:    {  17:        if (wf == null)  18:        {  19:            throw new IllegalArgumentException( "factory cannot be null" );  20:        }  21:        this.factory = wf;  22:    }  23:      24:    public void writeToFile( String location, String text )  25:        throws IOException  26:    {  27:        if (location == null) return;  28:        if (!location.equals( this.openLocation ))  29:        {  30:            if (this.w != null)  31:            {  32:                --this.openCount;  33:                this.w.close();  34:            }  35:            ++this.openCount;  36:            this.w = this.factory.createWriter( location );  37:        }  38:        this.w.write( text );  39:    }  40:      41:    public int getNumberWritersOpen()  42:    {  43:        return this.openCount;  44:    }  45:}

Obviously, this class isn't very thread safe - w could easily beclosed in one thread when a thread execution switch causes another threadto run its write statement on the closed writer, or even writing to adifferent writer than what was created (this is data corruption, which isworse than a raised exception).

Note that this class is designed for testing: we can easily create streams inour tests and note what gets written and to which stream. This follows apattern of designing for mock objects.

Beginning the Unit Test Class

The test class starts off with the standard naming conventions and constructor,and all necessary imports:

 1:import net.sourceforge.groboutils.junit.v1.MultiThreadedTestRunner; 2:import net.sourceforge.groboutils.junit.v1.TestRunnable; 3:import junit.framework.TestCase; 4:import junit.framework.TestSuite; 5:import java.io.*; 6:import java.util.*; 7: 8:public class WriterCacheUTest extends TestCase 9:{10:    public WriterCacheUTest( String name )11:    {12:        super( name );13:    }

Next, we need a way to archive the WriterCache created streams in ourownIWriterFactory:

15:    public static class MyWriterFactory implements WriterCache.IWriterFactory16:    {17:        Hashtable nameToStream = new Hashtable();18:        public Writer createWriter( String location )19:            throws IOException20:        {21:            StringWriter sw = (StringWriter)nameToStream.get( location );22:            if (sw == null)23:            {24:                sw = new StringWriter();25:                nameToStream.put( location, sw );26:            }27:            return sw;28:        }29:    }30:}

It uses StringWriter instances so that the tests can examine whatwhat written to which stream.

Ignoring the standard non-threaded tests for such a class (for brevity's sake),we move onto the threaded tests. But before we can write the tests, we need todevise the different tests that we can perform.

1. One thread can write to one location with a known text pattern, while another writes to a different location with a different text pattern. The test would check to make sure that each location contains only the expected text pattern, and exactly the number of text patterns actually written. Let's call this test "TextPattern"
2. Use lots of writing threads (say 10), but this time we check for the number of opened writers to ensure it never becomes negative, and generate some kind of warning if it is not 0 or 1. Let's call this test "OpenWriterCount".

Note: since this class uses a StringWriter instance as thereturnedWriter, the WriterCache calls to close()on the instance will be ignored, as this is the documented behavior of theStringWriter class. Thus, the sameStringWriter instancecan be sent to the WriterCache over and over.

Creating a TestRunnable

In order to support the above tests, we'll need a way to generate the textpattern for a given location, a given number of times. This is where theGroboUtil extension for multi-threaded testing comes into play. We createanother inner class to generate theWriterCache calls by extendingthe TestRunnable class:

31:    static class WriteText extends TestRunnable32:    {33:        private WriterCache wc;34:        private int count;35:        private String textPattern;36:        private String location;37:        private int sleepTime;38:        public WriteText( WriterCache wc, int count, String pattern,39:            String loc, int delay )40:        {41:            this.wc = wc;42:            this.count = count;43:            this.textPattern = pattern;44:            this.location = loc;45:            this.sleepTime = delay;46:        }47:        48:        public void runTest() throws Throwable49:        {50:            for (int i = 0; i < this.count; ++i)51:            {52:                Thread.sleep( this.sleepTime );53:                this.wc.writeToFile( this.location, this.textPattern );54:            }55:        }56:    }57:}

The void runTest() method must be implemented by concrete subclasses ofTestRunnable: this is the equivalent of the standard Java interfaceRunnable'svoid run() method, but properly wrapped fortesting.

Running Several Tasks In Parallel

This allows us to begin writing the test for "TextPattern". This test says thatwe need to write to two different streams with different text in parallel.Then, we need to ensure that the correct amount of data was written to each,and that no inappropriate data was written. In this test, we'll have oneWriteText instance write to location "0" with text "0" (we'll call thisthread 0), and another write to location "1" with text "1" (we'll call thisthread 1). To vary things up, thread 0 will write 10 times with a 50millisecond wait, and thread 1 will write 12 times with a 20 millisecond wait.The test ends up looking like:

59:    public void testTextPattern() throws Throwable60:    {61:        MyWriterFactor mwf = new MyWriterFactory();62:        WriterCache wc = new WriterCache( mwf );63:        TestRunnable tcs[] = {64:                new WriteText( wc, 10, "0", "0", 50 ),65:                new WriteText( wc, 12, "1", "1", 20 )66:            };67:        MultiThreadedTestRunner mttr =68:            new MultiThreadedTestRunner( tcs );69:        mttr.runTestRunnables( 2 * 60 * 1000 );70:        String s0 = mwf.nameToStream.get( "0" ).toString();71:        String s1 = mwf.nameToStream.get( "1" ).toString();72:        assertEquals( "Data corruption: stream 0", 10,73:            s0.length() );74:        assertEquals( "Data corruption: stream 1", 12,75:            s1.length() );76:        assertTrue( "Incorrect data written to stream 0.",77:            s0.indexOf( '1' ) < 0 );78:        assertTrue( "Incorrect data written to stream 1.",79:            s1.indexOf( '0' ) < 0 );80:    }81:

Lines 61-62 initialize the WriterCache instance with our test factory.Line 63 creates our list of parallel tasks to execute, which are instances ofourWriteText class above.

Line 64 creates an instance of theMultiThreadedTestRunner class, a utility class which handles thecreation, execution, and termination of the threads which run theTestRunnable instances. Line 65 invokes the utility method to run thetasks. The argument to this method specifies the maximum amountof time (in milliseconds) to let the threads run before killing them and markingthe execution as a failure (throughjunit.famework.Assert). If anyof the TestRunnable instances dies due to an exception (including anAssert failure), then all of the running threads are terminated, and therunTestRunnables method rethrows the underlying exception.

The remainder of the test ensures that only the correct data was placed whereit was intended to go.

Parallel Monitoring of Threaded Access

Now we can move on to the OpenWriterCount test. As of GroboTestingJUnitversion 1.2.0, there is a very simple way to add a thread that monitors thestatus of the common object while other threads manipulate its state.

The monitors are in a separate group from the runners, as the monitors areintended to loop over a set of checks until all the runners have completed.Before this functionality was added, the monitor runners had to communicatewith the other runners to learn when the runners were finished. Now, theMultiThreadedTestRunner class handles this communication.

We create another runnable class, but this time subclassing fromTestMonitorRunnable. This frees the new class from having to performthe proper looping and detection of when the runners have completed.In order to use this added functionality, the subclasses overload therunMonitor() method instead of the runTest() method.

83:    public static class WriterCountMonitor extends TestMonitorRunnable84:    {85:        private WriterCache wc;86:        public WriterCountMonitor( WriterCache wc )87:        {88:            this.wc = wc;89:        }90:        91:        public void runMonitor() throws Throwable92:        {93:            int open = this.wc.getNumberWritersOpen();94:            assertTrue(95:                "Invalid number of open writers.",96:                open == 0 || open == 1 );97:        }98:    }99:

The monitors are added to the MultiThreadedTestRunner througha second argument in another constructor. For our test, in order to havehigh confidence that threading errors causing the writer open count tobe invalid (not 1 or 0), we need to have a sufficient number of threadedaccess on the object-under-test. So, we generate 30 runners to iterate500 times each over the writer.

101:    public void testOpenWriterCount() throws Throwable102:    {103:        int runnerCount = 30;104:        int iterations = 500;105:        MyWriterFactor mwf = new MyWriterFactory();106:        WriterCache wc = new WriterCache( mwf );107:        TestRunnable tcs[] = new TestRunnable[ runnerCount ];108:        for (int i = 0; i < runnerCount; ++i)109:        {110:            tcs[i] = new WriteText( wc, 500, ""+(char)i,111:                ""+(char)i, 50 );112:        }113:        TestRunnable monitors[] = {114:                new WriterCountMonitor( wc );115:            };116:        MultiThreadedTestRunner mttr =117:            new MultiThreadedTestRunner( tcs, monitors );118:        mttr.runTestRunnables( 10 * 60 * 1000 );119:        120:        // verify streams121:        for (int i = 0; i < runnerCount; ++i)122:        {123:            String s = mwf.nameToString.get( ""+(char)i ).toString();124:            assertEquals( "Data corruption: stream "+i,125:                500, s.length() );126:        }127:    }128:

So, while this test runs the 30 runners, each writing 500 times, themonitor runs concurrently, analyzing the status of theWriterCacheinstance to ensure its integrety.

As one would expect, both of these tests expose serious synchronizationflaws in theWriterCache implementation.

Things To Look Out For

This package isn't without its pitfalls. Here's a checklist to reviewto ensure that your tests comply with the MTTR caveats.

1. Never directly run TestRunnable instances. They are designed to only be run in threads generated by therunTestRunnables() method inside the MultiThreadedTestRunner class.
2. The TestRunnable subclasses need to have their runTest() methods be watchful ofInterruptedException and Thread.currentThread().isInterrupted(). TheMultiThreadedTestRunner prematurely halts the runner threads by calling Thread.interrupt() on them. If the threads don't terminate themselves within a certain time limit, thenMultiThreadedTestRunner will perform the dangerous Thread.stop() operation, in order to prevent threads from going rogue. (Future versions may allow theThread.stop() call to be disabled.)

Alternatives

One alternative design to this approach is Greg Vaughn's TestDecoratorapproach, which is locatedhere. However, it forces the decorated Test instance to create thethreads itself, and does nothing to manage runaway threads.