Behavior counters for improved JUnit tests

A weak unit test may give false assurance if it is not responsive to API and test data changes.

Introduction
Recently I came across and even contributed some unit tests that were low quality. Not that they did not test what we wanted. When certain classes were changed the tests did not break, they seemed to still be testing the original API. Since I can’t duplicate the actual tests in this blog post, I’ll try to make up something very simple to illustrate the issue.

Example
In listing one below, a Revenue class is designed to merely illustrate the concept. The class contains a simple list of revenue per salesperson, for example.

Listing 1, example SUT

package com.octodecillion.junit;

import java.util.ArrayList;
import java.util.List;

public class Revenue {
   private List<Integer> users;

   public List<Integer> getUsers() {
     return users;
     // future bad change:
     //return new ArrayList<Integer>();
   }

   public void setUsers(List<Integer> users) {
    this.users = users;
   }
}

In listing 2 below we access this list and assert that each revenue is positive. Yeah, doesn’t make any business sense, bear with me.

Listing 2, A JUnit test of the Revenue class

package com.octodecillion.junit;

import static org.junit.Assert.*;
import java.util.ArrayList;
import org.junit.Before;
import org.junit.Test;

/**  */
public class RevenueTest {
	private Revenue revenue;
	private ArrayList<Integer> list;

	@Before
	public void setUp() throws Exception {
	 list = new ArrayList<Integer>();
	 revenue = new Revenue();
	}

	@Test
	public void should_have_positive_revenue() {
	  list.add(25);
	  list.add(99);
	  revenue.setUsers(list);
	  	
	  for (Integer tax : revenue.getUsers()) {
	    assertTrue(tax > 0);
	  }
	}
}

Problem
Listing 2 above will correctly test the Revenue class. But, what would happen if the Revenue’s class getUsers() method is accidentally changed to return a new List that has no entries? The test as written will still pass! Since the list has no entries the for loop will be skipped and the assertion will not be executed. We can protect against this by having a test of the size of the list, like so:

assertTrue(revenue.getUsers().size() > 0);

But, in a complex test there may be a limit on how extensive is the proactive testing. Or thru lack of skill or error the test could have been written incorrectly. Worse case, unit or integration tests are bogus, and the real problems will be found in production. What could be done?

Writing tests for the tests would be a “turtles all the way” scenario. We need to put inline behavioral coverage in the tests. By contrast, in normal behavioral testing with Mock Objects, the system under test (SUT) will have the behavioral testing, for example, Behavior-based testing with JMockit.

Solution
Listing 3 below, shows an alternative: we use an execution counter. For each block that must be executed within the unit test, not within the SUT, we increment a counter. At the end of the test we assert that the counter has a specific value. This ensures that no block is inadvertently skipped due to SUT changes or test data values.

Listing 3, JUnit test using execution counter

package com.octodecillion.junit;

import static org.junit.Assert.*;
import java.util.ArrayList;
import org.junit.Before;
import org.junit.Test;

public class RevenueTest {
	private Revenue revenue;
	private ArrayList<Integer> list;
	private InvokeCounter invokeCounter= new InvokeCounter();

	@Before
	public void setUp() throws Exception {
	  list = new ArrayList<Integer>();
	  revenue = new Revenue();
	}

	@Test
	public void should_have_positive_revenue() {
	  list.add(25);
	  list.add(99);
	  revenue.setUsers(list);
		
	  for (Integer tax : revenue.getUsers()) {
	    invokeCounter.increment();
	    assertTrue(tax > 0);
	  }	
		
	  invokeCounter.assertCounterMin(1);
	}

}

Implementation
Listing 4 below is possible implementation of an invocation counter class. This was updated (2012-11-19) to allow named counters.

Listing 4, Invocation counter implementation Gist

Click to expand source

Alternatives
The main alternative to the above technique is Of course, writing better tests. Another alternative is to use code coverage reporting for the JUnit tests. In a ‘test infected’ organization this would work. The reports are scrutinized and thresholds set and so forth. Alas, this is a passive approach or relies too much on manual intervention.

Summary
Discussed was a possible weakness of unit tests. An approach using invocation counters to assert block coverage of a test was presented.

Links

1. Code Coverage
2. JUnit
3. junit.org seems to be down
4. JUnit wikipedia entry
5. JMockit
6. Unit Testing

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.