Performing an action when a test fails

July 8, 2012

Performing an action when a test fails

Filed under: Automation, Java, Programming, Selenium, TDD, Test Automation — Tags: @ClassRule, @Rule, JUnit, Screen shot, Screen shot on failure, Selenium, Test automation, Web app testing, WebDriver — Thomas Sundberg @ 17:08

This post is available at http://www.thinkcode.se/blog/2012/07/08/performing-an-action-when-a-test-fails

Comments (34)

34 Comments »

This is awesome stuff and really helped me with re-writing out test suite. I’d like to log a failure using log4j with the test method name in the log, any idea how I can pass it to the “SimpleOnFailed” object so it can pass it to the logger?

Comment by Chris Fort — July 26, 2012 @ 20:22

Reply
- Hi!
  
  I would consider implementing the SimpleOnFailed class as:
```
package se.waymark;

import org.apache.log4j.Logger;
import org.junit.rules.TestWatcher;
import org.junit.runner.Description;

public class SimpleOnFailed extends TestWatcher {
    private static Logger log = Logger.getLogger(SimpleOnFailed.class);

    @Override
    protected void failed(Throwable e, Description description) {
        log.info(description.getClassName());
        log.info(description.getMethodName());
        log.info(description.getDisplayName());
    }
}
```
  By explorig the description parameter, you should find the method name of the failed method and be able to pass it to a logger.
  
  HTH
  Thomas
  
  Comment by Thomas Sundberg — July 27, 2012 @ 06:36
  
  Reply
  - Works great, and now I have a new rabbit hole to dive into. Thanks again!
    
    Comment by Chris Fort — July 27, 2012 @ 15:13
In my test suite @After , we do log out to the system and @Afterclass We close and quite the webdriver and browser. I want to use the @Rule to take screenshot of any specific failed method. I used above code and other code example and found out driver take screen shot after log out to the system. Just wanted to know how would i run @Rule before @After to a junit test class.

Comment by Reaz Patwry — August 22, 2012 @ 20:57

Reply
- A rule should only be executed when an exception has been thrown. Is an exception thrown in your @After method? If you introduce an error in a test, will the rule be executed then? If this doesn’t trigger an execution of the rule, then you have some problem with your rule implementation or the test setup.
  
  I would suggest that you implement a very small test, with a very simple rule and verifies that you get this setup to execute the rule. I would then extend the example when it works as expected and try to take a screenshot.
  
  HTH
  Thomas
  
  Comment by Thomas Sundberg — August 27, 2012 @ 12:57
  
  Reply
Excellent post Thomas! This is exactly what I need to do to make my test suite more sophisticated.

Comment by Amit Sett — October 14, 2012 @ 09:15

Reply
- I am glad you enjoyed it!
  
  /Thomas
  
  Comment by Thomas Sundberg — October 14, 2012 @ 09:17
  
  Reply
I would like to save the screenshot with the name of the Test Method that failed. What is the best way to do this?

Comment by Amit Sett — October 15, 2012 @ 01:08

Reply
- I would explore the Description class that is sent to the failed() method as a parameter. I would probably use the method getMethodName(), getDisplayName(), getClassName() or combination of them. The Javadoc may tell you more http://kentbeck.github.com/junit/javadoc/latest/org/junit/runner/Description.html
  
  /Thomas
  
  Comment by Thomas Sundberg — October 15, 2012 @ 06:29
  
  Reply
I have one doubt,, let’s say i am creating a driver object in @Before, so how can i pass the driver object to @Rule
public ScreenShotRule screenShootRule = new ScreenShotRule(browser); so my base test looks like this
@Rule
public ScreenShotRule screenShootRule = new ScreenShotRule(browser);
@Before
public void doSetup(){
//Here i am instantiating driver object.
}

So how do i pass driver object to ScreenShot class?

Thanks,
Jeevan.

Comment by g1reddyclrJeevan — January 25, 2013 @ 00:54

Reply
- Hi!
  
  I see two options for setting a driver in the rule.
  
  1) Include it in the constructor and construct the rule instance in the @Before method with the constructor that requires the Driver object. This is what I essentially does in the screen shot example, but I create the browser (driver) before each test instead. JUnit creates all it’s instance variables before each test is executed and the result is the same as if you do it in a before method.
  
  2) Add a setter to the rule object and inject the driver object in the rule.
  
  Either one of these solutions will work.
  
  The last alternative could look like this in code:
```
@Rule
public SimpleOnFailed ruleExample = new SimpleOnFailed();

@Before
public void doSetup() {
    Object driver = new Object();
    ruleExample.setDriver(driver);
}
```
  HTH
  Thomas
  
  Comment by Thomas Sundberg — January 25, 2013 @ 06:42
  
  Reply
  - Hi Thomas,
    
    What are the pro’s/con’s of each approach?
    
    -Amit
    
    Comment by Amit Sett — February 1, 2013 @ 02:40
  - If you mean the two different ways I describe how you could set a driver in a rule, then it is more a matter of taste than anything else.
    
    I tend to think that an object should be possible to use directly after it has been constructed. This means supplying all dependencies that are mandatory on the constructor. I would probably have used the first option since it would be harder for me to forget to set the driver. It is sometimes called CDI, Constructor Dependency Injection.
    
    HTH
    Thomas
    
    Comment by Thomas Sundberg — February 1, 2013 @ 16:52
There’s one problem with this approach. The method annotated with @After is executed before Rule#failed() is invoked. In @After we always log out from the application (login in @Before). Hence, if a test fails we always get the same screen shot – that of the login form (shown after logout).

Comment by frightanic — February 13, 2013 @ 23:51

Reply
- Hi!
  
  If the after hook breaks the possibility to take a good screenshot then I assume that you will have to find another way to do this? an option could be to have tests that delegates to other methods to do the actual clicking and possible the verification and catches their exceptions. When an exception occurs, take a screenshot. This would not use the built in rule mechanism in JUnit, but it might solve your problem.
  
  I am not sure that this would be a great solution. But it could be a sufficiently good solution.
  
  Thomas
  
  Comment by Thomas Sundberg — February 16, 2013 @ 11:02
  
  Reply
If I do something like this:

@Rule
public TestWatcher testWatchThis = new TestWatcher() {
@Override
protected void failed(Throwable e, Description description) {
LOG.error(“Only executed when a test fails”);
}
};

within a test class, I always get an Exception complaining that “Field testWatchThis must implement MethodRule”.

What’s the deal with that? I’m using JUnit v4.11.

Comment by MPP — May 29, 2013 @ 22:01

Reply
- Hi!
  
  TestWatcher is abstract, at least in JUnit 4.10. Implement your own rule that extends TestWatcher instead. I implement a simple example in the class SimpleOnFailed.
  
  HTH
  Thomas
  
  Comment by Thomas Sundberg — May 30, 2013 @ 06:33
  
  Reply
Hi,

My test file is as follows :

import org.junit.Rule;
import org.junit.Test;
import selenium.utils.ui.OpenBrowserOnFailure;

import static org.junit.Assert.assertTrue;

public class SampleTest
{
@Rule
public OpenBrowserOnFailure ruleExample = new OpenBrowserOnFailure();

@Test
public void shouldFail()
{
assertTrue(false);
}
}

and my Rule contains following code:

import org.junit.rules.TestWatcher;
import org.junit.runner.Description;

public class OpenBrowserOnFailure extends TestWatcher
{
public OpenBrowserOnFailure()
{
System.out.println(“in constructor”);
}

@Override
protected void failed(Throwable e, Description description)
{
System.out.println(“Only executed when a test fails”);
}
}

I get output “in constructor” but the overridden method failed() is not being called. What have I done wrong?

Thanks in advance!

Comment by 1234 — August 21, 2013 @ 19:02

Reply
Hi!

I can’t really say why it doesn’t work for you. I copied your example and pasted it into a test class I called SampleTest exactly as below. It worked for me. I use JUnit 4.11. I didn’t create two public classes, I put everything in the same class.

import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TestWatcher;
import org.junit.runner.Description;

import static org.junit.Assert.assertTrue;

public class SampleTest {
@Rule
public OpenBrowserOnFailure ruleExample = new OpenBrowserOnFailure();

@Test
public void shouldFail() {
assertTrue(false);
}

@Test
public void shouldPass() {
assertTrue(true);
}
}

class OpenBrowserOnFailure extends TestWatcher {
public OpenBrowserOnFailure() {
System.out.println(“in constructor”);
}

@Override
protected void failed(Throwable e, Description description) {
System.out.println(“Only executed when a test fails”);
}
}

We seem to be using the same imports, the same names etc. The output from an execution for me looks like this:

in constructor
Only executed when a test fails

java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at SampleTest.shouldFail(SampleTest.java:14)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:77)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:195)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)

in constructor

It tells me that the constructor has been called twice and that the tests failed once, just as expected.

Sorry, I can’t spot your error.
Thomas

Comment by Thomas Sundberg — August 21, 2013 @ 21:13

Reply
- Hi Thomas,
  
  Thanks for replying. The code seems to be working when the test file is ran directly from command line. But I am using ant in my project. It doesn’t work when I call the class from within a target in build.xml. Do you think ant would have anything to do with this?
  
  Thanks again for the quick response. Would really appreciate your input.
  
  Thanks!
  
  Comment by 1234 — August 24, 2013 @ 01:25
  
  Reply
  - Hi,
    
    I figured it out. The issue was with my build.xml. Junit runner.JUnitCore was not in my classpath.
    
    Thanks for the time and consideration.
    
    Thanks! 🙂
    
    Comment by 1234 — August 24, 2013 @ 02:29
Hi!

I am glad that it worked out for you.

It sounded like a path problem and obviously were as well. This is one of the reasons why I don’t use Ant for building Java apps anymore. Ant is great for handling files and building custom setups, copying files and so forth. It is not so great for handling dependencies and the class path. I use Maven for that.

/Thomas

Comment by Thomas Sundberg — August 24, 2013 @ 08:44

Reply
Hi Thomas. Great article, i find it really useful. So i got rid of all the @Before and @After annotations in my test classes and i am using testrules that implement the TestWatcher as you suggested.
Anyway – do you have any idea when these “failed” and “finished” are executed? They seem to be fired virtually simultaneously, so once in every 10-20 turns, i get an exception that the driver is already disposed, hence it cannot take a screenshot. Any ideas how to get over this? And, while debugging, i get it always.

Stack

java.lang.AssertionError: There were no rows to click on, aborting…
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:26)

….
org.openqa.selenium.remote.SessionNotFoundException: The FirefoxDriver cannot be used after quit() was called.
Build info: version: ‘2.33.0’, revision: ‘4e90c97’, time: ‘2013-05-22 15:33:32’
System info: os.name: ‘Windows 7’, os.arch: ‘amd64’, os.version: ‘6.1’, java.version: ‘1.7.0_21’
Driver info: driver.version: FirefoxDriver
at org.openqa.selenium.firefox.FirefoxDriver$LazyCommandExecutor.execute(FirefoxDriver.java:352)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:527)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:569)
at org.openqa.selenium.firefox.FirefoxDriver.getScreenshotAs(FirefoxDriver.java:316)
at PikoFuncTests.TestHelpers.TestRules.failed(TestRules.java:47)
at org.junit.rules.TestWatcher.failedQuietly(TestWatcher.java:84)
at org.junit.rules.TestWatcher.access$300(TestWatcher.java:46)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:62)

Comment by E. — August 29, 2013 @ 13:28

Reply
- Hi!
  
  It is hard to really say what the problem is without looking at your code.
  
  My gut feeling tells me that it sounds like a concurrency problem. There is more then one thread involved and you have some kind of race condition that the test rules looses every 10 – 20th time. And as far as I understand every time when you do it manually. I would look at where you destroy the browser and see if it would be possible to remove it at the absolute latest possible moment. Suppose that you have a global browser instance, when is it destroyed? It seems to be destroyed earlier than you had hoped for.
  
  Sorry that I can’t be more specific.
  Thomas
  
  Comment by Thomas Sundberg — August 29, 2013 @ 14:29
  
  Reply
  - Hi. It actually seems to be failing everytime at the moment. It feels like the finished method is called before failed method.
    I have created a question in stackoverflow with my issue with code samples. It can be found here: http://stackoverflow.com/questions/18510854/junit-testwatcher-failed-and-finished-methods-firing-time
    I would be nice if you could look into it.
    
    Comment by E. — August 29, 2013 @ 15:14
Hi!

I looked at your problem at Stackoverflow and realized that your problem is the shared resource and how it gets destroyed.

I created a small example where I was able to show it. Let me start with a happy flow:

package se.think.code;

import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TestWatcher;
import org.junit.runner.Description;

import java.util.LinkedList;
import java.util.List;

import static org.junit.Assert.assertTrue;

public class ComplicatedRuleExampleTest {
@Rule
public TestRules testRules = new TestRules();

@Test
public void shouldPass() throws Exception {
assertTrue(true);
}

@Test
public void shouldFail() throws Exception {
// assertTrue(false);
}

@Test
public void shouldAlsoFail() throws Exception {
// assertTrue(false);
}
}

class TestRules extends TestWatcher {
private static List sharedResource;

public TestRules() {
System.out.println(“constructor for instance ” + this);
if (sharedResource == null) {
sharedResource = SharedResource.get();
}
}

@Override
protected void starting(Description description) {
System.out.println(“starting”);
sharedResource = SharedResource.get();
}

@Override
protected void failed(Throwable e, Description description) {
System.out.println(“failed, we need to use the shared resource”);
System.out.println(“the shared resource has the size ” + sharedResource.size());
}

@Override
protected void finished(Description description) {
System.out.println(“finished”);
SharedResource.clean();
}
}

class SharedResource {
private static List shared = new LinkedList();

public static List get() {
return shared;
}

public static void clean() {
shared = null;
}
}

Executing all tests will result in this:

constructor for instance se.think.code.TestRules@61905335
starting
finished
constructor for instance se.think.code.TestRules@9a4d5c6
starting
finished
constructor for instance se.think.code.TestRules@24cebf1c
starting
finished

Your problem is that the rule is an instance variable that is created every time and that it contains a shared resource that is not created fresh every time. Your shared resource is cleaned at the finished method and will not be available when it should be used later. The example above does not show that. But if we enable the two methods that should fail, then we will get a result similar to the result below. The order between the stack traces may differ between different executions.

constructor for instance se.think.code.TestRules@76dbab83
starting
failed, we need to use the shared resource
the shared resource has the size 0
finished

java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)

starting
failed, we need to use the shared resource
finished

java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)

java.lang.NullPointerException
at se.think.code.TestRules.failed(ComplicatedRuleExampleTest.java:52)
at org.junit.rules.TestWatcher.failedQuietly(TestWatcher.java:84)

constructor for instance se.think.code.TestRules@1c61a885
starting
finished
constructor for instance se.think.code.TestRules@4f7ec251

java.lang.NullPointerException
at se.think.code.TestRules.failed(ComplicatedRuleExampleTest.java:52)
at org.junit.rules.TestWatcher.failedQuietly(TestWatcher.java:84)

The solution must be not to clear the shared resource in the finish method. Instead just clear it at the end of the test class using an after class annotation, @AfterClass

HTH
Thomas

Comment by Thomas Sundberg — August 30, 2013 @ 06:25

Reply
- Alright, thank you very much. I solved the problem by implementing the “sharedResource” as a instance variable.
  
  Comment by E. — August 30, 2013 @ 10:37
  
  Reply
[…] https://thomassundberg.wordpress.com/2012/07/08/performing-an-action-when-a-test-fails/ […]

Pingback by Ahondando en Junit: Avisando sólo de los errores. | Adictos al Trabajo — August 31, 2015 @ 09:59

Reply
is there any way to take screenshot in python using same code?

Comment by Tanvi — March 31, 2016 @ 14:39

Reply
- I am sure that it is. I have never done it.
  
  This could be an inspiration:
  
  from selenium import webdriver
  
  driver = webdriver.Remote(“http://localhost:4444/wd/hub”, webdriver.DesiredCapabilities.FIREFOX.copy())
  driver.get(“http://www.google.com”)
  driver.get_screenshot_as_file(‘/Screenshots/google.png’)
  
  Found at http://www.seleniumhq.org/docs/04_webdriver_advanced.jsp and using Python as programming language.
  
  Cheers,
  Thomas
  
  Comment by Thomas Sundberg — March 31, 2016 @ 15:21
  
  Reply
hi

Am Getting the Below Error

java.lang.Exception.Field screenshotRule must implement Method Rule

Code :

public class ScreenshotTest

{
private WebDriver driver;

@Rule
public ScreenshotRule screenshotRule = new ScreenshotRule(driver);

@Test
public void shouldfail()
{
System.setProperty(“webdriver.chrome.driver”,”C://Driver//chromedriver.exe”);
driver = new ChromeDriver();
driver.get(“https://www.google.co.in/#gfe_rd=cr&gws_rd=cr,ssl”);
driver.findElement(By.linkText(“Esakki”));

}
}

public class ScreenshotRule extends TestWatcher

{

private WebDriver driver;

public ScreenshotRule (WebDriver driver)
{
this.driver = driver;
}

@Override
protected void failed(Throwable e , Description description)
{
System.out.println(“Am Taking Screenshot Using the Method”);
}

@Override
protected void finished(Description description)
{
driver.close();
}

}

Please Help Me Friends , Whats the error ? What am i missing

Comment by Esakki — June 3, 2016 @ 12:30

Reply
- Hi!
  
  It seems like you have not implemented a method in your class ScreenshotRule. It is possible that you are using another version of your dependencies than I did when I wrote the example.
  
  Look at what methods TestWatcher defines and override the methods you want to use.
  
  HTH
  Thomas
  
  Comment by Thomas Sundberg — June 4, 2016 @ 21:57
  
  Reply
Hello there. I’m struggling with the problem that few guys already mentioned. So, your code is great, but i still have great issue when using @After annotation ( which in my case i need to use to logout user). As mention before, problem is that overriden method “failed” is invoced after @After method. Do you have some new ideas how this should be solved?

Kind regards,
Djole

Comment by Djole — July 7, 2016 @ 21:25

Reply
Great Job! It helped me. Thanks!

Comment by Pilar — August 9, 2016 @ 19:30

Reply