Test Coverage Using JMockit

The JMockit Unit Testing library continues to astound. One new thing I discovered is its Coverage reporting.

Code Coverage
Code coverage is simply a measurement of what code has been actually run when tests are executed. There are many such measures, ramifications, and tools. Like testing itself, code coverage measurement is probably not done enough, or misused.

“Code coverage tells you what you definitely haven’t tested, not what you have.” — Mark Simpson in comment

Path Coverage
Plenty of coverage reporting tools out there. What this one also includes is Path coverage. This is different then branch coverage. Paths are possible execution paths from entry points to exit points. If you visualize a methods statements in a directed graph, paths are a enumeration of the possible edges traversed when that method is invoked. So, Path coverages is inclusive of Branch coverage. Well, I’m not a testing expert, so this may be way off.

Very surprising results. For example, you run a coverage report with a tool such as Cobertura or Emma and feel very happy that you exercised every line and branch with your tests. Then you run the same tests but use JMockit Coverage and discover your tests didn’t cover all the paths! Not only that your line coverage wasn’t so great either.

Report
JMockit explicitly gives you a report showing:

Path
Measures how many of the possible execution paths through method/constructor bodies were actually executed by tests.
The percentages are calculated as 100*NPE/NP, where NP is the number of possible paths and NPE the number of fully executed paths.

Line
Measures how much of the executable production code was exercised by tests. An executable line of code contains one or more executable segments.
The percentages are calculated as 100*NE/NS, where NS is the number of segments and NE the number of executed segments.

Data
Measures how many of the instance and static non-final fields were fully exercised by the test run. To be fully exercised, a field must have the last value assigned to it read by at least one test. The percentages are calculated as 100*NFE/NF, where NF is the number of non-final fields and NFE the number of fully exercised fields.

– from the JMockit coverage report HTML page

Other information is found by using the full HTML output option.

Example
A sample JMockit coverage report is here. Of course you can drill down into various parts of the html page. Like when you click on an exercised line you will get a list of what invoked that line.

Worth it?
Are the metrics such as Path coverage that this tool generates accurate? Is JMockit coverage a replacement for other tools such as Cobertura? I don’t know. For most projects, the resources would probably make the use of coverages generated by multiple tools prohibitive.

Evaluation
One possible approach to evaluating coverage tools is to just use actual real results of the target application. Use the list of bugs and correlate to a coverage tool report. Where were the bugs? Which tool gave the least measure for this location?

Further Reading

• Pitfalls of code coverage
• Measuring line and path coverage with JMockit Coverage
• What’s the point of basis path coverage?
• Basis-Testing
• Code coverage tools in Java
• Statement, Branch, and Path Coverage Testing in Java
• FLOWGRAPHS AND PATH TESTING
• What is Wrong with Statement Coverage
• Easy code coverage reports with JMockIt
• Unit Test Code Coverage
• Planning for JMockit 2