Test coverage – friend or foe?

Measuring the test coverage is something many people think is a good idea. But is it a good idea? It depends on the quality of the tests. Test coverage may be a good measurement if the tests are good. But suppose that we have a high degree of test coverage and bad tests?

I will show two examples of how to get a 100% test coverage using Cobertura. They will be based on a set of good tests and a set of bad tests.

How is test coverage calculated?

Test coverage is calculated by recording which lines of code that has been executed. If the code has been executed through a test in a testing framework then we have calculated the test coverage.

The calculation is done using these steps:

• Start with instrumenting compiled code
• Execute the instrumented code
• Each execution of a line is recorded in a log
• Combining this execution log with the source code enables us to calculate how many lines out of the total number of lines that has been executed

We will be able to say that 47% of the lines in the source code has been executed. If the execution is done through test code, this will give us a measurement of the test coverage.

What is a good test?

A colleague asked that question the other day. What is a good test? It is a really difficult question to answer. There are a number of properties that hold true for a good test. I will list some of them and discuss why I think they are important. Others may think that the order of importance is different or that other properties are more important then those that I list.

I will show some examples using Java and JUnit of what I think are bad tests and how they can be refactored to something better.

Tests are automated in my world. I always try to avoid manually testing. The main reason for this is that manual testing is slow and hard to repeat arbitrary many times. Therefore, test should be read as automated test in for the rest of this text.