Java Code Coverage: Cobertura vs. Emma vs Clover

Posted onOctober 23, 2010bytborthwick

The last time I looked for free java code coverage tools, I found EclEmma for Eclipse and Cobertura for automated builds through ant or maven. They’re both good tools, but they come up with slightly different numbers. Maybe I just missed it before, but there’s now an Eclipse plugin for Cobertura and amaven plugin for Emma. Both have Hudson and even Sonar plugins, so I thought I’d spend a little more time comparing them. Just for kicks, I also looked at Atlassian’s Clover even though it’s fairly expensive (but free for open-source projects).

Units of coverage

Cobertura focuses on lines and branches (ie. if and case statements); EclEmma concentrates on bytecode instructions, though you can also get line metrics. Emma has a good write-up of theirmethodology, but I didn’t see as much for Cobertura. Clover looks at statements, branches, and methods to come up witha total score. All these approaches result in fairly similar coverage results (unless your code does things like put multiple  statements on the same line separated by semi-colons) but they’ll almost always be a little different.

Eclipse plugin

All the tools will highlight covered and uncovered code in Eclipse for you. The Emma and Clover plugins have a nice graphic display and let you drill down.

Emma:

Clover:

Cobertura’s display in Eclipse is a bit weak, no drill-down into classes and no visual graph of coverage:

Full report

Probably more important than the Eclipse view are the reports that each tool can create, eg. in an automated build. I didn’t take much time to explore different configuration options here, just took the defaults for each.

Emma’s report actually has less visual appeal than its Eclipse plugin. If you invest some time, you can probably make it look better but this is what you get out of the box:

Cobertura, on the other hand, has some nice graphics. If you just configure report output as xml, and turn on the Hudson plugin, it looks like:

And here’s Clover:

The Emma and Cobertura reports include metrics on classes, lines, and methods. Cobertura also has conditional statements and files while Emma includes blocks. Conditionals could be interesting, I guess, if you notice a big coverage gap. Clover’s ‘elements’ appears to be lines and methods. Cobertura has nice drill-downs and Clover has not just drill-downs but tree map and word cloud charts as well.

Aside from graphics, I also looked a little closer at how some specific code elements are handled.

Static nested classes

The code under test for the result page below has an internal builder class. If you’ve never noticed, the compiler creates an extra $1 class for you behind the scenes. Emma reports the SampleBuilder$1 class as uncovered:

So somehow you have 100% coverage of blocks, lines, and methods, but only 67% coverage of classes… Cobertura excludes SampleBuilder$1 in its class coverage to give you 100%, but if you drill-down, you see the class with N/A in its line and method coverage columns, which seems like a good solution. Clover does not show the $1 class at all, which is also fine.

Enums

Emma dings you if your tests don’t explicitly test the built-in enum valueOf andvalues() methods- they show up as 0% coverage. Like with the inner classes, Cobertura will show those methods but mark them as N/A and not include them in your method coverage percentage. Clover will not show them at all.

Annotations

Clover will show you an annotation (an @interface that you’ve created) with N/A. It won’t appear at all in Emma. It also won’t appear in Clover, but curiosly, I did notice it shows up in a word cloud chart of top project risks.

Default no-arg constructor

If your class doesn’t define its own constructor, but a test calls the implicit no-arg constructor, does it count in the coverage? For Emma, yes. It lists the constructor as classname(), eg.Sample() and gives you credit for covering one method and one line. Cobertura will show the constructor as <init>() but give it an N/A for coverage. Clover won’t show an undeclared default constructor at all.

Conclusion

These are all good tools. In Eclipse, I’m mostly concerned with finding untested lines and getting a general sense of how much code is covered so I think I’ll keep using EclEmma. For a full report, Cobertura has a nicer look and fewer coverage oddities to note. Also, even though new plugins have come out recently for Emma, it doesn’t appear that themain project is still being actively developed. As far as speed goes, we try to keep our unit tests very fast anyway, so it was not a big concern and I didn’t bother investigating it.

Clover looks goods visually, has lots of options, and I like how it handles results. If money were no object, I’d definitely look more closely at it, but for our situation, the free tools work just fine