Functional Testing: lessons learned

Finally I got around using functional testing (some people call it  ‘System Testing‘) extensively in a project. This post lists what I’ve learned about functional testing and more specifically, Geb, which allows you to automate functional tests.

Let’s start with defining what I understand by ‘functional test': a functional test is run against a fully running web application, which is on a dedicated environment (more on that later). Functional tests are a great asset in your quality assurance arsenal because they add to the confidence you have in your system and you will catch yet more bugs early. Unit test- and integration tests are nice and necessary, but I think they are subservient to the result of functional tests which finally determine the quality of the system.

Functional tests don’t replace manual testing

Very complex scenario’s may be quite hard (e.g. expensive) to automate, so it’s probably best you leave those to your testing team. The best candidates for automating are:

  • scenario that cover notoriously buggy bits of functionality
  • simple scenario’s that are mind-numbing to test for the testing team

Talk to your testers and decide together which scenarios you want to automate. They will know.

Geb is great

I scanned the functional testing tool framework landscape, and Selenium quickly emerged. Using Selenium IDE is not an option if you want to have a maintainable, robust test suite. Looking onwards, writing Selenium tests in Java didn’t really appeal to me either. Another tricky API to learn, lots of Java to write… until I found Geb. In essence, Geb is a Groovy DSL for telling a browser what to do. It is built on top of Selenium. Until now, Geb isn’t very well-known but there is no reason why that shouldn’t change quickly. The Spock/Gradle/Geb sample is where I got started.

Geb creates screenshots and html dumps of every test it runs, to speed op diagnosing tests that have failed. It plays nicely with Hudson / Jenkins in a sense that it reports test results in jUnit style. What you will need to do is setup a Windows box as the machine that runs all your tests. If your application has a history of suffering from bugs that occur in one browser but not in another, you will want to run the tests in different browsers, probably including Internet Explorer. Since it allows installing all major browsers, a Windows box is a natural choice. On that box you configure a Jenkins slave (through Java Web Start) that kicks off the tests.

Running automatically is tricky

You’ll need to have some mechanism in place that resets your system under test to a default state before a test runs. This is harder to do than with unit tests: you’ll have to clear databases, messages queues and whatever other resources your system uses. This will take time. Make sure you take this into account before putting a lot of effort into developing functional tests.

Functional tests take time to run

Unlike unit tests, which are very quick, functional tests are actual user scenario’s, so they can take quite some time to run. This can become a real issue when you have a lot of tests. So keep this in mind when you add tests and when you decide which scenarios to automate. Maybe you’ll need to split them up further, maybe you need to adjust the granularity of your scenario. It all depends :)

Mix in other types of system clients

This post focusses on giving input to the system through a browser, but there is no reason why your Geb-based test can not also talk to some kind of webservice that your system exposes. As long as the scenario is realistic, it is valid. What is not valid is to peek inside your system’s database to check for result. The rule is: if the system doesn’t allow a user (including machine clients) to do it, you can’t do it in your test. That would dilute the value of your test.

The end result

Me and the team on my project added about 50 tests for a system we were having trouble with to get under control. Geb allowed us to that quite quickly. I did some pioneering myself, and after that it was quite easy to get other team members on board. Soon enough, everyone was coding tests for scenarios they knew needed to be automated. After a few weeks of the test suite showing up as green in Jenkins (or blue if you’re unfortunate enought not to have the Green Balls plugin), the project had left the rough seas it had been in. For us, that was a big win :)

, , , ,

3 Responses to Functional Testing: lessons learned

  1. Soubhagya October 6, 2012 at 1:31 am #

    Hi ,
    I am trying to automate a web application using Geb/spock but stuck with the following error

    Have u ever come across such a situation?
    I am using FF 10 and Selenium 2.25

    org.openqa.selenium.WebDriverException: Permission denied for to get property HTMLDocument.compatMode
    Command duration or timeout: 14 milliseconds
    Build info: version: ‘2.25.0’, revision: ‘17482’, time: ‘2012-07-18 22:18:01′
    System info: os.name: ‘Windows 7′, os.arch: ‘x86′, os.version: ‘6.1’, java.version: ‘1.6.0_20′
    Driver info: driver.version: RemoteWebDriver
    Session ID: 0e3c9628-521f-4ae5-98b5-216d2a027648
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:188)
    at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:145)
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:498)
    at org.openqa.selenium.remote.RemoteWebElement.execute(RemoteWebElement.java:244)
    at org.openqa.selenium.remote.RemoteWebElement.click(RemoteWebElement.java:77)
    at org.openqa.selenium.WebElement$click.call(Unknown Source)
    at geb.navigator.NonEmptyNavigator.click(NonEmptyNavigator.groovy:297)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1047)
    at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
    at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:149)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
    at geb.content.TemplateDerivedPageContent.click(TemplateDerivedPageContent.groovy:105)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1047)
    at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
    at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:149)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
    at NikeDotNetSpec.$spock_feature_2_0(NikeDotNetSpec.groovy:73)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.spockframework.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:176)
    at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:316)
    at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:297)
    at org.spockframework.runtime.BaseSpecRunner.invokeFeatureMethod(BaseSpecRunner.java:284)
    at org.spockframework.runtime.BaseSpecRunner.doRunIteration(BaseSpecRunner.java:255)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.spockframework.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:176)
    at org.spockframework.runtime.extension.MethodInvocation.invokeTargetMethod(MethodInvocation.java:91)
    at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:85)
    at org.spockframework.runtime.extension.builtin.AbstractRuleInterceptor$1.evaluate(AbstractRuleInterceptor.java:37)
    at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:46)
    at org.spockframework.runtime.extension.builtin.TestRuleInterceptor.intercept(TestRuleInterceptor.java:38)
    at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:84)
    at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:305)
    at org.spockframework.runtime.BaseSpecRunner.runIteration(BaseSpecRunner.java:222)
    at org.spockframework.runtime.BaseSpecRunner.initializeAndRunIteration(BaseSpecRunner.java:214)
    at org.spockframework.runtime.BaseSpecRunner.runSimpleFeature(BaseSpecRunner.java:204)
    at org.spockframework.runtime.BaseSpecRunner.doRunFeature(BaseSpecRunner.java:198)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.spockframework.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:176)
    at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:316)
    at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:297)
    at org.spockframework.runtime.BaseSpecRunner.runFeature(BaseSpecRunner.java:174)
    at org.spockframework.runtime.BaseSpecRunner.runFeatures(BaseSpecRunner.java:151)
    at org.spockframework.runtime.BaseSpecRunner.doRunSpec(BaseSpecRunner.java:111)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.spockframework.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:176)
    at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:316)
    at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:297)
    at org.spockframework.runtime.BaseSpecRunner.runSpec(BaseSpecRunner.java:90)
    at org.spockframework.runtime.BaseSpecRunner.run(BaseSpecRunner.java:81)
    at org.spockframework.runtime.Sputnik.run(Sputnik.java:63)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:76)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:195)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
    Caused by: org.openqa.selenium.remote.ErrorHandler$UnknownServerException: Permission denied for to get property HTMLDocument.compatMode
    Build info: version: ‘2.25.0’, revision: ‘17482’, time: ‘2012-07-18 22:18:01′
    System info: os.name: ‘Windows 7′, os.arch: ‘x86′, os.version: ‘6.1’, java.version: ‘1.6.0_20′
    Driver info: driver.version: unknown

    Process finished with exit code 0

Trackbacks/Pingbacks

  1. An Army of Solipsists » Blog Archive » This Week in Grails (2012-13) - April 2, 2012

    […] Functional Testing: lessons learned […]

  2. Questa settimana in Grails (2012-13) - luca-canducci.com - Il blog di Luca Canducci: notizie, tips e nuove tecnologie dal mondo dell’IT. - April 3, 2012

    […] Functional Testing: lessons learned […]

Leave a Reply