Thomas Sundberg

July 8, 2012

Performing an action when a test fails

This post is available at http://www.thinkcode.se/blog/2012/07/08/performing-an-action-when-a-test-fails

Advertisements

34 Comments »

  1. This is awesome stuff and really helped me with re-writing out test suite. I’d like to log a failure using log4j with the test method name in the log, any idea how I can pass it to the “SimpleOnFailed” object so it can pass it to the logger?

    Comment by Chris Fort — July 26, 2012 @ 20:22

    • Hi!

      I would consider implementing the SimpleOnFailed class as:

      package se.waymark;
      
      import org.apache.log4j.Logger;
      import org.junit.rules.TestWatcher;
      import org.junit.runner.Description;
      
      public class SimpleOnFailed extends TestWatcher {
          private static Logger log = Logger.getLogger(SimpleOnFailed.class);
      
          @Override
          protected void failed(Throwable e, Description description) {
              log.info(description.getClassName());
              log.info(description.getMethodName());
              log.info(description.getDisplayName());
          }
      }
      

      By explorig the description parameter, you should find the method name of the failed method and be able to pass it to a logger.

      HTH
      Thomas

      Comment by Thomas Sundberg — July 27, 2012 @ 06:36

      • Works great, and now I have a new rabbit hole to dive into. Thanks again!

        Comment by Chris Fort — July 27, 2012 @ 15:13

  2. In my test suite @After , we do log out to the system and @Afterclass We close and quite the webdriver and browser. I want to use the @Rule to take screenshot of any specific failed method. I used above code and other code example and found out driver take screen shot after log out to the system. Just wanted to know how would i run @Rule before @After to a junit test class.

    Comment by Reaz Patwry — August 22, 2012 @ 20:57

    • A rule should only be executed when an exception has been thrown. Is an exception thrown in your @After method? If you introduce an error in a test, will the rule be executed then? If this doesn’t trigger an execution of the rule, then you have some problem with your rule implementation or the test setup.

      I would suggest that you implement a very small test, with a very simple rule and verifies that you get this setup to execute the rule. I would then extend the example when it works as expected and try to take a screenshot.

      HTH
      Thomas

      Comment by Thomas Sundberg — August 27, 2012 @ 12:57

  3. Excellent post Thomas! This is exactly what I need to do to make my test suite more sophisticated.

    Comment by Amit Sett — October 14, 2012 @ 09:15

  4. I would like to save the screenshot with the name of the Test Method that failed. What is the best way to do this?

    Comment by Amit Sett — October 15, 2012 @ 01:08

  5. I have one doubt,, let’s say i am creating a driver object in @Before, so how can i pass the driver object to @Rule
    public ScreenShotRule screenShootRule = new ScreenShotRule(browser); so my base test looks like this
    @Rule
    public ScreenShotRule screenShootRule = new ScreenShotRule(browser);
    @Before
    public void doSetup(){
    //Here i am instantiating driver object.
    }

    So how do i pass driver object to ScreenShot class?

    Thanks,
    Jeevan.

    Comment by g1reddyclrJeevan — January 25, 2013 @ 00:54

    • Hi!

      I see two options for setting a driver in the rule.

      1) Include it in the constructor and construct the rule instance in the @Before method with the constructor that requires the Driver object. This is what I essentially does in the screen shot example, but I create the browser (driver) before each test instead. JUnit creates all it’s instance variables before each test is executed and the result is the same as if you do it in a before method.

      2) Add a setter to the rule object and inject the driver object in the rule.

      Either one of these solutions will work.

      The last alternative could look like this in code:

      @Rule
      public SimpleOnFailed ruleExample = new SimpleOnFailed();
      
      @Before
      public void doSetup() {
          Object driver = new Object();
          ruleExample.setDriver(driver);
      }
      

      HTH
      Thomas

      Comment by Thomas Sundberg — January 25, 2013 @ 06:42

      • Hi Thomas,

        What are the pro’s/con’s of each approach?

        -Amit

        Comment by Amit Sett — February 1, 2013 @ 02:40

      • If you mean the two different ways I describe how you could set a driver in a rule, then it is more a matter of taste than anything else.

        I tend to think that an object should be possible to use directly after it has been constructed. This means supplying all dependencies that are mandatory on the constructor. I would probably have used the first option since it would be harder for me to forget to set the driver. It is sometimes called CDI, Constructor Dependency Injection.

        HTH
        Thomas

        Comment by Thomas Sundberg — February 1, 2013 @ 16:52

  6. There’s one problem with this approach. The method annotated with @After is executed before Rule#failed() is invoked. In @After we always log out from the application (login in @Before). Hence, if a test fails we always get the same screen shot – that of the login form (shown after logout).

    Comment by frightanic — February 13, 2013 @ 23:51

    • Hi!

      If the after hook breaks the possibility to take a good screenshot then I assume that you will have to find another way to do this? an option could be to have tests that delegates to other methods to do the actual clicking and possible the verification and catches their exceptions. When an exception occurs, take a screenshot. This would not use the built in rule mechanism in JUnit, but it might solve your problem.

      I am not sure that this would be a great solution. But it could be a sufficiently good solution.

      Thomas

      Comment by Thomas Sundberg — February 16, 2013 @ 11:02

  7. If I do something like this:

    @Rule
    public TestWatcher testWatchThis = new TestWatcher() {
    @Override
    protected void failed(Throwable e, Description description) {
    LOG.error(“Only executed when a test fails”);
    }
    };

    within a test class, I always get an Exception complaining that “Field testWatchThis must implement MethodRule”.

    What’s the deal with that? I’m using JUnit v4.11.

    Comment by MPP — May 29, 2013 @ 22:01

    • Hi!

      TestWatcher is abstract, at least in JUnit 4.10. Implement your own rule that extends TestWatcher instead. I implement a simple example in the class SimpleOnFailed.

      HTH
      Thomas

      Comment by Thomas Sundberg — May 30, 2013 @ 06:33

  8. Hi,

    My test file is as follows :

    import org.junit.Rule;
    import org.junit.Test;
    import selenium.utils.ui.OpenBrowserOnFailure;

    import static org.junit.Assert.assertTrue;

    public class SampleTest
    {
    @Rule
    public OpenBrowserOnFailure ruleExample = new OpenBrowserOnFailure();

    @Test
    public void shouldFail()
    {
    assertTrue(false);
    }
    }

    and my Rule contains following code:

    import org.junit.rules.TestWatcher;
    import org.junit.runner.Description;

    public class OpenBrowserOnFailure extends TestWatcher
    {
    public OpenBrowserOnFailure()
    {
    System.out.println(“in constructor”);
    }

    @Override
    protected void failed(Throwable e, Description description)
    {
    System.out.println(“Only executed when a test fails”);
    }
    }

    I get output “in constructor” but the overridden method failed() is not being called. What have I done wrong?

    Thanks in advance!

    Comment by 1234 — August 21, 2013 @ 19:02

  9. Hi!

    I can’t really say why it doesn’t work for you. I copied your example and pasted it into a test class I called SampleTest exactly as below. It worked for me. I use JUnit 4.11. I didn’t create two public classes, I put everything in the same class.

    import org.junit.Rule;
    import org.junit.Test;
    import org.junit.rules.TestWatcher;
    import org.junit.runner.Description;

    import static org.junit.Assert.assertTrue;

    public class SampleTest {
    @Rule
    public OpenBrowserOnFailure ruleExample = new OpenBrowserOnFailure();

    @Test
    public void shouldFail() {
    assertTrue(false);
    }

    @Test
    public void shouldPass() {
    assertTrue(true);
    }
    }

    class OpenBrowserOnFailure extends TestWatcher {
    public OpenBrowserOnFailure() {
    System.out.println(“in constructor”);
    }

    @Override
    protected void failed(Throwable e, Description description) {
    System.out.println(“Only executed when a test fails”);
    }
    }

    We seem to be using the same imports, the same names etc. The output from an execution for me looks like this:

    in constructor
    Only executed when a test fails

    java.lang.AssertionError
    at org.junit.Assert.fail(Assert.java:86)
    at org.junit.Assert.assertTrue(Assert.java:41)
    at org.junit.Assert.assertTrue(Assert.java:52)
    at SampleTest.shouldFail(SampleTest.java:14)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:77)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:195)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)

    in constructor

    It tells me that the constructor has been called twice and that the tests failed once, just as expected.

    Sorry, I can’t spot your error.
    Thomas

    Comment by Thomas Sundberg — August 21, 2013 @ 21:13

    • Hi Thomas,

      Thanks for replying. The code seems to be working when the test file is ran directly from command line. But I am using ant in my project. It doesn’t work when I call the class from within a target in build.xml. Do you think ant would have anything to do with this?

      Thanks again for the quick response. Would really appreciate your input.

      Thanks!

      Comment by 1234 — August 24, 2013 @ 01:25

      • Hi,

        I figured it out. The issue was with my build.xml. Junit runner.JUnitCore was not in my classpath.

        Thanks for the time and consideration.

        Thanks! 🙂

        Comment by 1234 — August 24, 2013 @ 02:29

  10. Hi!

    I am glad that it worked out for you.

    It sounded like a path problem and obviously were as well. This is one of the reasons why I don’t use Ant for building Java apps anymore. Ant is great for handling files and building custom setups, copying files and so forth. It is not so great for handling dependencies and the class path. I use Maven for that.

    /Thomas

    Comment by Thomas Sundberg — August 24, 2013 @ 08:44

  11. Hi Thomas. Great article, i find it really useful. So i got rid of all the @Before and @After annotations in my test classes and i am using testrules that implement the TestWatcher as you suggested.
    Anyway – do you have any idea when these “failed” and “finished” are executed? They seem to be fired virtually simultaneously, so once in every 10-20 turns, i get an exception that the driver is already disposed, hence it cannot take a screenshot. Any ideas how to get over this? And, while debugging, i get it always.

    Stack

    java.lang.AssertionError: There were no rows to click on, aborting…
    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:26)

    ….
    org.openqa.selenium.remote.SessionNotFoundException: The FirefoxDriver cannot be used after quit() was called.
    Build info: version: ‘2.33.0’, revision: ‘4e90c97’, time: ‘2013-05-22 15:33:32’
    System info: os.name: ‘Windows 7’, os.arch: ‘amd64’, os.version: ‘6.1’, java.version: ‘1.7.0_21’
    Driver info: driver.version: FirefoxDriver
    at org.openqa.selenium.firefox.FirefoxDriver$LazyCommandExecutor.execute(FirefoxDriver.java:352)
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:527)
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:569)
    at org.openqa.selenium.firefox.FirefoxDriver.getScreenshotAs(FirefoxDriver.java:316)
    at PikoFuncTests.TestHelpers.TestRules.failed(TestRules.java:47)
    at org.junit.rules.TestWatcher.failedQuietly(TestWatcher.java:84)
    at org.junit.rules.TestWatcher.access$300(TestWatcher.java:46)
    at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:62)

    Comment by E. — August 29, 2013 @ 13:28

    • Hi!

      It is hard to really say what the problem is without looking at your code.

      My gut feeling tells me that it sounds like a concurrency problem. There is more then one thread involved and you have some kind of race condition that the test rules looses every 10 – 20th time. And as far as I understand every time when you do it manually. I would look at where you destroy the browser and see if it would be possible to remove it at the absolute latest possible moment. Suppose that you have a global browser instance, when is it destroyed? It seems to be destroyed earlier than you had hoped for.

      Sorry that I can’t be more specific.
      Thomas

      Comment by Thomas Sundberg — August 29, 2013 @ 14:29

  12. Hi!

    I looked at your problem at Stackoverflow and realized that your problem is the shared resource and how it gets destroyed.

    I created a small example where I was able to show it. Let me start with a happy flow:

    package se.think.code;

    import org.junit.Rule;
    import org.junit.Test;
    import org.junit.rules.TestWatcher;
    import org.junit.runner.Description;

    import java.util.LinkedList;
    import java.util.List;

    import static org.junit.Assert.assertTrue;

    public class ComplicatedRuleExampleTest {
    @Rule
    public TestRules testRules = new TestRules();

    @Test
    public void shouldPass() throws Exception {
    assertTrue(true);
    }

    @Test
    public void shouldFail() throws Exception {
    // assertTrue(false);
    }

    @Test
    public void shouldAlsoFail() throws Exception {
    // assertTrue(false);
    }
    }

    class TestRules extends TestWatcher {
    private static List sharedResource;

    public TestRules() {
    System.out.println(“constructor for instance ” + this);
    if (sharedResource == null) {
    sharedResource = SharedResource.get();
    }
    }

    @Override
    protected void starting(Description description) {
    System.out.println(“starting”);
    sharedResource = SharedResource.get();
    }

    @Override
    protected void failed(Throwable e, Description description) {
    System.out.println(“failed, we need to use the shared resource”);
    System.out.println(“the shared resource has the size ” + sharedResource.size());
    }

    @Override
    protected void finished(Description description) {
    System.out.println(“finished”);
    SharedResource.clean();
    }
    }

    class SharedResource {
    private static List shared = new LinkedList();

    public static List get() {
    return shared;
    }

    public static void clean() {
    shared = null;
    }
    }

    Executing all tests will result in this:

    constructor for instance se.think.code.TestRules@61905335
    starting
    finished
    constructor for instance se.think.code.TestRules@9a4d5c6
    starting
    finished
    constructor for instance se.think.code.TestRules@24cebf1c
    starting
    finished

    Your problem is that the rule is an instance variable that is created every time and that it contains a shared resource that is not created fresh every time. Your shared resource is cleaned at the finished method and will not be available when it should be used later. The example above does not show that. But if we enable the two methods that should fail, then we will get a result similar to the result below. The order between the stack traces may differ between different executions.

    constructor for instance se.think.code.TestRules@76dbab83
    starting
    failed, we need to use the shared resource
    the shared resource has the size 0
    finished

    java.lang.AssertionError
    at org.junit.Assert.fail(Assert.java:86)
    at org.junit.Assert.assertTrue(Assert.java:41)

    starting
    failed, we need to use the shared resource
    finished

    java.lang.AssertionError
    at org.junit.Assert.fail(Assert.java:86)
    at org.junit.Assert.assertTrue(Assert.java:41)

    java.lang.NullPointerException
    at se.think.code.TestRules.failed(ComplicatedRuleExampleTest.java:52)
    at org.junit.rules.TestWatcher.failedQuietly(TestWatcher.java:84)

    constructor for instance se.think.code.TestRules@1c61a885
    starting
    finished
    constructor for instance se.think.code.TestRules@4f7ec251

    java.lang.NullPointerException
    at se.think.code.TestRules.failed(ComplicatedRuleExampleTest.java:52)
    at org.junit.rules.TestWatcher.failedQuietly(TestWatcher.java:84)

    The solution must be not to clear the shared resource in the finish method. Instead just clear it at the end of the test class using an after class annotation, @AfterClass

    HTH
    Thomas

    Comment by Thomas Sundberg — August 30, 2013 @ 06:25

    • Alright, thank you very much. I solved the problem by implementing the “sharedResource” as a instance variable.

      Comment by E. — August 30, 2013 @ 10:37

  13. is there any way to take screenshot in python using same code?

    Comment by Tanvi — March 31, 2016 @ 14:39

    • I am sure that it is. I have never done it.

      This could be an inspiration:

      from selenium import webdriver

      driver = webdriver.Remote(“http://localhost:4444/wd/hub”, webdriver.DesiredCapabilities.FIREFOX.copy())
      driver.get(“http://www.google.com”)
      driver.get_screenshot_as_file(‘/Screenshots/google.png’)

      Found at http://www.seleniumhq.org/docs/04_webdriver_advanced.jsp and using Python as programming language.

      Cheers,
      Thomas

      Comment by Thomas Sundberg — March 31, 2016 @ 15:21

  14. hi

    Am Getting the Below Error

    java.lang.Exception.Field screenshotRule must implement Method Rule

    Code :

    public class ScreenshotTest

    {
    private WebDriver driver;

    @Rule
    public ScreenshotRule screenshotRule = new ScreenshotRule(driver);

    @Test
    public void shouldfail()
    {
    System.setProperty(“webdriver.chrome.driver”,”C://Driver//chromedriver.exe”);
    driver = new ChromeDriver();
    driver.get(“https://www.google.co.in/#gfe_rd=cr&gws_rd=cr,ssl”);
    driver.findElement(By.linkText(“Esakki”));

    }
    }

    public class ScreenshotRule extends TestWatcher

    {

    private WebDriver driver;

    public ScreenshotRule (WebDriver driver)
    {
    this.driver = driver;
    }

    @Override
    protected void failed(Throwable e , Description description)
    {
    System.out.println(“Am Taking Screenshot Using the Method”);
    }

    @Override
    protected void finished(Description description)
    {
    driver.close();
    }

    }

    Please Help Me Friends , Whats the error ? What am i missing

    Comment by Esakki — June 3, 2016 @ 12:30

    • Hi!

      It seems like you have not implemented a method in your class ScreenshotRule. It is possible that you are using another version of your dependencies than I did when I wrote the example.

      Look at what methods TestWatcher defines and override the methods you want to use.

      HTH
      Thomas

      Comment by Thomas Sundberg — June 4, 2016 @ 21:57

  15. Hello there. I’m struggling with the problem that few guys already mentioned. So, your code is great, but i still have great issue when using @After annotation ( which in my case i need to use to logout user). As mention before, problem is that overriden method “failed” is invoced after @After method. Do you have some new ideas how this should be solved?

    Kind regards,
    Djole

    Comment by Djole — July 7, 2016 @ 21:25

  16. Great Job! It helped me. Thanks!

    Comment by Pilar — August 9, 2016 @ 19:30


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: