Test coverage – friend or foe?

Measuring the test coverage is something many people think is a good idea. But is it a good idea? It depends on the quality of the tests. Test coverage may be a good measurement if the tests are good. But suppose that we have a high degree of test coverage and bad tests?

I will show two examples of how to get a 100% test coverage using Cobertura. They will be based on a set of good tests and a set of bad tests.

How is test coverage calculated?

Test coverage is calculated by recording which lines of code that has been executed. If the code has been executed through a test in a testing framework then we have calculated the test coverage.

The calculation is done using these steps:

• Start with instrumenting compiled code
• Execute the instrumented code
• Each execution of a line is recorded in a log
• Combining this execution log with the source code enables us to calculate how many lines out of the total number of lines that has been executed

We will be able to say that 47% of the lines in the source code has been executed. If the execution is done through test code, this will give us a measurement of the test coverage.

Cucumber-JVM – not just for testing GUIs

Cucumber is a tool that supports Behaviour Driven Development, BDD. A lot of people think that the only place where a system has behaviour is in the user interface and especially in the graphical user interface. As a developer I know that this is not the case. All systems have behaviour at different places and different levels.

I will show an example of how a system can be developed using its desired behaviour and start from a non graphical point. I will work from the model down to the database and when I’m happy with the logical behaviour I will add a graphical user interface on top of it. I will actually add a few different interfaces; two web-based, one swing and two different types of web services. The result will be an example of Model View Controller, MVC, developed using BDD.

An important point when I add the GUIs or web services is that I will not change the desired behaviour. I will only change how the behaviour is verified. This is one way of showing you that Cucumber and BDD is not about testing GUIs. It is about systems behaviour.

Building the model

Previous – Introduction

The feature I will start with looks like this:

src/test/resources/se/waymark/rentit/Rent.feature

Feature: Rental cars should be possible to rent to gain revenue to the rental company.

  As an owner of a car rental company
  I want to make cars available for renting
  So I can make money

  Scenario: Find and rent a car
    Given there are 18 cars available for rental
    When I rent one
    Then there will only be 17 cars available for rental

It consists of three parts:

• Given – the preconditions of the system under test. The setup of the systems state if you want. In this case make 18 compact cars available for rental in the system.
• When – the actual change of the system. Transforming it from the initial state to the final state. Rent one car.
• Then – the expected final state of the system. The verification that the state change was the desired change. After one car is rented, there should only be 17 left to rent.

A JSF web application

Previous – Building the model

Many modern applications are built as web applications. The benefits are obvious, you don’t need to package your software in shrink-wrap and send it to your customers. Upgrading is easy, you have to upgrade the server you host the system on and that’s it.

The first user interface I will add to the rental system will therefore be a web GUI. It will be the simplest possible solution and the goal is not to build a fancy web app. The goal is to show how Cucumber can control a tool like Selenium WebDriver to assert the behaviour of the web application.

A Wicket web application

Previous – A JSF web application

A wicket application is yet another web application. I divide the project in two parts as earlier. The only large difference is the support class that will connect to the system under test. It has been adapted for another web application.

Building a Swing GUI

Previous – A Wicket web application

A Java Swing application is yet another graphical user interface that can be attached on top of the model developed earlier. The project is divided in the same way as earlier, in two parts. The only large difference here is the support class. It need to be adapted for a Swing user interface. Another difference is of obviously that the GUI is developed using Swing. But that has actually a rather small impact on the verification.

A RESTFul Web Service

Previous – A Swing application

A RESTFul web service is yet another way to use the model. It doesn’t have any user interface and it is expected to be used by third party suppliers. I divide the project using two Maven modules as earlier. One for the production code and one for the verification. The only large difference here is the support class that will connect to the system under test but now has to be adapted for a RESTFul web service instead of any graphical interface. Another difference is of course that there is no GUI and that it is developed using a Jersey servlet instead, but that has little impact on the verification.

A SOAP Web Service

Previous – A RESTFul web service

A SOAP Web Service is yet another way to use the model. It doesn’t have any user interface and is expected to be used by third party suppliers. I divide the project using two Maven modules as earlier. One for the production code and one for the verification. The only large difference here is the support class that will connect to the system under test but now has to be adapted for a SOAP web service instead of any graphical interface or RESTFul web service. Another difference is of course that there is no GUI and that it is developed using a CXF instead, but that has little impact on the verification.

Conclusion

Previous – A SOAP web service

I have shown you five different ways to implement something on top of a common model using the same behaviour. I hope you can agree that it is obvious that BDD and Cucumber-JVM is not just meant for testing GUIs. It can be used to assert any behaviour.

Test drive an implementation using an Executable Specification – revisited

An example is perhaps the best way to describe something. Concrete examples are easier to understand than abstract descriptions.

I will show how Cucumber-JVM can be used to specify an example and how the example can be connected to the system under test, SUT. The example can then be executed using any IDE or using Maven.