Finally I got around using functional testing (some people call it ’System Testing‘) extensively in a project. This post lists what I’ve learned about functional testing and more specifically, Geb, which allows you to automate functional tests.
Let’s start with defining what I understand by ‘functional test’: a functional test is run against a fully running web application, which is on a dedicated environment (more on that later). Functional tests are a great asset in your quality assurance arsenal because they add to the confidence you have in your system and you will catch yet more bugs early. Unit test- and integration tests are nice and necessary, but I think they are subservient to the result of functional tests which finally determine the quality of the system.
Functional tests don’t replace manual testing
Very complex scenario’s may be quite hard (e.g. expensive) to automate, so it’s probably best you leave those to your testing team. The best candidates for automating are:
- scenario that cover notoriously buggy bits of functionality
- simple scenario’s that are mind-numbing to test for the testing team
Talk to your testers and decide together which scenarios you want to automate. They will know.
Geb is great
I scanned the functional testing tool framework landscape, and Selenium quickly emerged. Using Selenium IDE is not an option if you want to have a maintainable, robust test suite. Looking onwards, writing Selenium tests in Java didn’t really appeal to me either. Another tricky API to learn, lots of Java to write… until I found Geb. In essence, Geb is a Groovy DSL for telling a browser what to do. It is built on top of Selenium. Until now, Geb isn’t very well-known but there is no reason why that shouldn’t change quickly. The Spock/Gradle/Geb sample is where I got started.
Geb creates screenshots and html dumps of every test it runs, to speed op diagnosing tests that have failed. It plays nicely with Hudson / Jenkins in a sense that it reports test results in jUnit style. What you will need to do is setup a Windows box as the machine that runs all your tests. If your application has a history of suffering from bugs that occur in one browser but not in another, you will want to run the tests in different browsers, probably including Internet Explorer. Since it allows installing all major browsers, a Windows box is a natural choice. On that box you configure a Jenkins slave (through Java Web Start) that kicks off the tests.
Running automatically is tricky
You’ll need to have some mechanism in place that resets your system under test to a default state before a test runs. This is harder to do than with unit tests: you’ll have to clear databases, messages queues and whatever other resources your system uses. This will take time. Make sure you take this into account before putting a lot of effort into developing functional tests.
Functional tests take time to run
Unlike unit tests, which are very quick, functional tests are actual user scenario’s, so they can take quite some time to run. This can become a real issue when you have a lot of tests. So keep this in mind when you add tests and when you decide which scenarios to automate. Maybe you’ll need to split them up further, maybe you need to adjust the granularity of your scenario. It all depends 
Mix in other types of system clients
This post focusses on giving input to the system through a browser, but there is no reason why your Geb-based test can not also talk to some kind of webservice that your system exposes. As long as the scenario is realistic, it is valid. What is not valid is to peek inside your system’s database to check for result. The rule is: if the system doesn’t allow a user (including machine clients) to do it, you can’t do it in your test. That would dilute the value of your test.
The end result
Me and the team on my project added about 50 tests for a system we were having trouble with to get under control. Geb allowed us to that quite quickly. I did some pioneering myself, and after that it was quite easy to get other team members on board. Soon enough, everyone was coding tests for scenarios they knew needed to be automated. After a few weeks of the test suite showing up as green in Jenkins (or blue if you’re unfortunate enought not to have the Green Balls plugin), the project had left the rough seas it had been in. For us, that was a big win
Hi ,
I am trying to automate a web application using Geb/spock but stuck with the following error
Have u ever come across such a situation?
I am using FF 10 and Selenium 2.25
org.openqa.selenium.WebDriverException: Permission denied for to get property HTMLDocument.compatMode
Command duration or timeout: 14 milliseconds
Build info: version: ’2.25.0′, revision: ’17482′, time: ’2012-07-18 22:18:01′
System info: os.name: ‘Windows 7′, os.arch: ‘x86′, os.version: ’6.1′, java.version: ’1.6.0_20′
Driver info: driver.version: RemoteWebDriver
Session ID: 0e3c9628-521f-4ae5-98b5-216d2a027648
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:188)
at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:145)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:498)
at org.openqa.selenium.remote.RemoteWebElement.execute(RemoteWebElement.java:244)
at org.openqa.selenium.remote.RemoteWebElement.click(RemoteWebElement.java:77)
at org.openqa.selenium.WebElement$click.call(Unknown Source)
at geb.navigator.NonEmptyNavigator.click(NonEmptyNavigator.groovy:297)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1047)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:149)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
at geb.content.TemplateDerivedPageContent.click(TemplateDerivedPageContent.groovy:105)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1047)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:149)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
at NikeDotNetSpec.$spock_feature_2_0(NikeDotNetSpec.groovy:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.spockframework.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:176)
at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:316)
at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:297)
at org.spockframework.runtime.BaseSpecRunner.invokeFeatureMethod(BaseSpecRunner.java:284)
at org.spockframework.runtime.BaseSpecRunner.doRunIteration(BaseSpecRunner.java:255)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.spockframework.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:176)
at org.spockframework.runtime.extension.MethodInvocation.invokeTargetMethod(MethodInvocation.java:91)
at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:85)
at org.spockframework.runtime.extension.builtin.AbstractRuleInterceptor$1.evaluate(AbstractRuleInterceptor.java:37)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:46)
at org.spockframework.runtime.extension.builtin.TestRuleInterceptor.intercept(TestRuleInterceptor.java:38)
at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:84)
at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:305)
at org.spockframework.runtime.BaseSpecRunner.runIteration(BaseSpecRunner.java:222)
at org.spockframework.runtime.BaseSpecRunner.initializeAndRunIteration(BaseSpecRunner.java:214)
at org.spockframework.runtime.BaseSpecRunner.runSimpleFeature(BaseSpecRunner.java:204)
at org.spockframework.runtime.BaseSpecRunner.doRunFeature(BaseSpecRunner.java:198)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.spockframework.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:176)
at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:316)
at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:297)
at org.spockframework.runtime.BaseSpecRunner.runFeature(BaseSpecRunner.java:174)
at org.spockframework.runtime.BaseSpecRunner.runFeatures(BaseSpecRunner.java:151)
at org.spockframework.runtime.BaseSpecRunner.doRunSpec(BaseSpecRunner.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.spockframework.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:176)
at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:316)
at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:297)
at org.spockframework.runtime.BaseSpecRunner.runSpec(BaseSpecRunner.java:90)
at org.spockframework.runtime.BaseSpecRunner.run(BaseSpecRunner.java:81)
at org.spockframework.runtime.Sputnik.run(Sputnik.java:63)
at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:76)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:195)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: org.openqa.selenium.remote.ErrorHandler$UnknownServerException: Permission denied for to get property HTMLDocument.compatMode
Build info: version: ’2.25.0′, revision: ’17482′, time: ’2012-07-18 22:18:01′
System info: os.name: ‘Windows 7′, os.arch: ‘x86′, os.version: ’6.1′, java.version: ’1.6.0_20′
Driver info: driver.version: unknown
Process finished with exit code 0