Test Case Selection 101 – Unit Test Edition

After we have finished this lesson, we

• Understand why we should not try to minimize the number of test cases.
• Know why most of our unit tests should test only one class.
• Understand why we should not write our code from the ground up.

Watch the Lesson

The text version of this lesson is given in the following:

Three Tips That Help Us to Select Our Test Cases

1. Don’t Try to Minimize the Number of Test Cases

When people write their first unit tests, they typically write test cases that try to assert too many things. For example, these test cases might:

• Verify the interactions that happened between the system under test and a mock.
• Ensure that the objects passed to the mock objects contain the correct information.
• Verify that the tested method returns the correct information.

For example, if we have to verify that the save() method of the TaskService class is working as expected, and we want to minimize the number of our unit tests, the source code of our test class might look as follows (the test case is highlighted):

@RunWith(HierarchicalContextRunner.class)
public class TaskServiceTest {
 
    private TaskRepository repository;
    private TaskService service;
     
    @Before
    public void configureSUT() {
        repository = mock(TaskRepository.class);
        service = new TaskService(repository);
    }
	
	public class Save {
	
		TaskFormDTO input;
	
		@Before
		public void configureTestCase() {
			input = createInput();
			returnCreatedTask(input);
		}
		
		private TaskFormDTO createInput() {
			TaskFormDTO input = new TaskFormDTO();
			input.setTitle("title");
			input.setDescription("description");
			return input;
		}
		
		private void returnCreatedTask(TaskFormDTO input) {
			Task created = new TaskBuilder()
					.withId(1L)
					.withCreator(4L)
					.withTitle("title")
					.withDescription("description")
					.withStatusOpen()
					.build();
			given(repository.save(input)).willReturn(created);
		}
		
		@Test
		public void shouldCreateNewTaskAndReturnInformationOfCreatedTask() {
			Task returned = service.save(input);
			
			verify(repository, times(1)).save(assertArg(t -> {
					assertThat(t.getTitle()).isEqualTo("title");
					assertThat(t.getDescription()).isEqualTo("description");
			}));
			
			assertThat(returned.getId()).isEqualByComparingTo(1L);
			
			assertThat(returned.getAssignee()).isNull();
			assertThat(returned.getCloser()).isNull();
			assertThat(returned.getCreator().getId())
					.isEqualByComparingTo(4L);
			
			assertThat(returned.getDescription()).isEqualTo("description");
			assertThat(returned.getTitle()).isEqualTo("title");
			
			assertThat(returned.getStatus()).isEqualTo(TaskStatus.OPEN);
			assertThat(returned.getResolution()).isNull();
		}
	}
}

This test case has three problems:

First, it is “impossible” to figure out short and descriptive name for it. A good test name must describe the expected result, but we cannot really do it because our test verifies too many things.

Second, our test case is hard to read and maintain. The problem is that our test doesn’t have a clear goal and that is why our test is a lot longer than it should be.

Third, our test case can fail for multiple reasons. One of the reasons why we write unit tests is that we want to either acquire or share information about the tested unit. The problem is that because our test can fail because of multiple reasons, we don’t necessarily get all the the information we need when it fails.

For example, if our test fails because of the id of the returned task is not correct, we have no idea if the other properties of the returned task are correct. It is possible that after we have fixed our code, our test case fails again because the other properties of the returned task are not correct. This is a total waste of time because we might have to run our test case multiple times before it passes.

It is clear that we shouldn’t write tests that have too many assertions. Some people claim that one unit test should have only one assertion. However, I think that it is a bit too extreme. Typically I follow the advice found from this comment (by Roy Osherove)

My guideline is usually that you test one logical CONCEPT per test. you can have multiple asserts on the same *object*. they will usually be the same concept being tested.

The benefit of this advice is that it helps us to write shorter unit tests that are easier to read, write, and maintain. Also, since our unit test tests only one logical concept, it is easier to figure out a good name for the test method.

For example, we could write the following unit tests for the save() method of the TaskService class:

• Verify that the tested method creates a new task by using the correct information.
• Verify that the returned task has the correct id.
• Ensure that the tested method returns a task that has the correct title.
• Ensure that the returned task has the correct description.
• Verify the returned task has no assignee.
• Verify that creator of the returned is correct.
• Ensure that the tested method returns an open task. This test ensures that the status of the returned task is TaskStatus.OPEN And that the created task has no resolution and closer.

After we have written these tests, the source code of our test class looks as follows:

@RunWith(HierarchicalContextRunner.class)
public class TaskServiceTest {
 
    private TaskRepository repository;
    private TaskService service;
     
    @Before
    public void configureSUT() {
        repository = mock(TaskRepository.class);
        service = new TaskService(repository);
    }
	
	public class Save {
	
		TaskFormDTO input;
	
		@Before
		public void configureTestCase() {
			input = createInput();
			returnCreatedTask(input);
		}
		
		private TaskFormDTO createInput() {
			TaskFormDTO input = new TaskFormDTO();
			input.setTitle("title");
			input.setDescription("description");
			return input;
		}
		
		private void returnCreatedTask(TaskFormDTO input) {
			Task created = new TaskBuilder()
					.withId(1L)
					.withCreator(4L)
					.withTitle("title")
					.withDescription("description")
					.withStatusOpen()
					.build();
			given(repository.save(input)).willReturn(created);
		}
		
		@Test
		public void shouldCreateNewTaskWithCorrectInformation() {
			service.save(input);
			
			verify(repository, times(1)).save(assertArg(t -> {
					assertThat(t.getTitle()).isEqualTo("title");
					assertThat(t.getDescription()).isEqualTo("description");
			}));
		}
		
		@Test
		public void shouldReturnTaskThatHasCorrectId() {
			Task returned = service.save(input);
		
			assertThat(returned.getId()).isEqualByComparingTo(1L);
		}
		
		@Test
		public void shouldReturnTaskThatHasCorrectTitle() {
			Task returned = service.save(input);
			
			assertThat(returned.getTitle()).isEqualTo("title");
		}
		
		@Test
		public void shouldReturnTaskThatHasCorrectDescription() {
			Task returned = service.save(input);
			
			assertThat(returned.getDescription()).isEqualTo("description");
		}
		
		@Test
		public void shouldReturnTaskThatHasNoAssignee() {
			Task returned = service.save(input);
			
			assertThat(returned.getAssignee()).isNull();
		}
		
		@Test
		public void shouldReturnTaskThatHasCorrectCreator() {
			Task returned = service.save(input);
			
			assertThat(returned.getCreator().getId())
					.isEqualByComparingTo(4L);			
		}
		
		@Test
		public void shouldReturnOpenTask() {
			Task returned = service.save(input);
			
			assertThat(returned.getCloser()).isNull();
			assertThat(returned.getStatus()).isEqualTo(TaskStatus.OPEN);
			assertThat(returned.getResolution()).isNull();
		}
	}
}

I think that it is pretty obvious that our new test class is a lot cleaner than the old one. However, there are a few test cases that have more than one assertion. This means that we might have to rerun them if these test cases fail.

We can solve this problem by using soft assertions. If you don’t what soft assertions are, you should take a look at the ‘Writing Assertions’ topic (this topic is available only for my customers).

2. Write Most Tests as Close to the Tested Code as Possible

One of the most controversial topics of unit testing is selecting the size of tested unit. There are two vocal groups that have very strong opinion about this. If we listen to these extremist groups, we have two options:

1. We can write tests which concentrate on testing one use case. For example, if we think about our task tracker, the tests that belong to this category ensure that the service method (or the controller method) which resolves a task is working as expected.
2. We can write tests which test one method. The tests that belong to this category ensure that the resolve() method of the Task class is working as expected.

The most obvious difference of these two testing strategies is that the first strategy uses a bigger unit size than the second strategy. This means that:

• The tests which belong to the first category help us to ensure that individual classes are working as expected when they are tested as a group.
• The tests which which belong to the second category help us to verify that the methods of a single class are working as expected.

I know that the tests which belong to the category one sound a lot like integration tests, but I want to make one thing absolutely clear:

There is no rule which states that a unit test must always test only one method of one class.

It is OK to write unit tests which use real objects instead of test doubles AND use test doubles for replacing objects that communicate with external systems such as databases. In fact, these tests are quite valuable because they help us to ensure that our application fulfills its requirements and they are fast to run because they don’t use external systems such as databases.

Nevertheless, I think that most of our unit tests should test only one class. I will explain my opinion by using a simple example.

First, if we want to ensure that the service method that resolves a task is working as expected, we have to:

1. Create the tested service object and its dependencies.
2. Create the Task object that will be closed.
3. Configure the repository mock to return the created Task object when the tested service method invokes the method that finds a task from the used database.
4. Invoke the service method.
5. Verify the expected result.

Second, if we want ensure that the resolve() method of the Task object is working as expected, we have to:

1. Create the Task object that will be closed.
2. Invoke the resolve() method.
3. Verify the expected result.

This simple example should demonstrate that the main reason why we should write most of our unit tests on class level is that these tests are a lot simpler than tests which test the same thing by invoking a service (or a controller) method. Because these tests require less code, they are easier to write, read, and maintain than tests which test complete use cases.

To summarize, we should write most of our unit tests on class level because this gives us the best return of investment. That being said, we should still write unit tests that concentrate on testing complete use cases, but these tests shouldn’t test every possible scenario because writing the required test code takes too long and makes our tests hard to read and maintain.

3. Write Code From Top to Bottom

When we start implementing a new feature, we either get a list of requirements that describe how the feature should work or we have create this list together with our customer. In any case, after we know the business requirements of the implemented feature, we naturally have to implement it.

We can implement basically any feature by using one of these two options:

1. We can build it from the ground up like we build a house. If we select this option, we have to finish our task in this order: create the required data model, implement the required repositories, write our service classes, write the required controllers, and implement the user interface.
2. We can build it from top to bottom. If we select this option, we have to finish our task in this order: implement the user interface, write the required controllers, write our service classes, create the required data model, and implement the required repositories.

At first it might seem that the only difference between these options is the order in which we implement the required components. However, these options have a more subtle difference that helps us to select the correct test cases.

• If we implement our feature from the ground up, it is often hard to link our test cases with the actual business requirements. In other words, it is hard to tell which business requirement is not fulfilled if a test case fails.
• If we implement our feature from top to bottom, the requirements of the upper layer defines the requirements for the layer directly below it. This means that we can transform these requirements into test cases which can be traced back to the actual business requirements.

I have noticed that some developers prefer to build features from the ground up. Actually, I used to do it too in the past. However, the problem was that writing tests for my code felt pointless and mentally draining. That is why I started to write my tests on auto pilot. This is not a very good idea because I ended up writing a lot of poorly written and unnecessary unit tests.

However, that all changed when I started to write my code from top to bottom. Because I can write my tests by using the requirements of the implemented component, I can avoid writing unnecessary unit tests and concentrate on writing tests that have a clear purpose. This makes my job a lot easier and helps to write less and better test code that reduces maintenance efforts as well.

Summary

This lesson has taught us three things:

• We should write small unit tests which test one logical concept.
• We should write most of our unit tests as close to the tested code as possible because these tests are easy to read, write, and maintain.
• We should write our code from top to bottom because this helps to us write tests that can be traced back to the actual business requirements.