Feeds

Agitating Java and testing Windows

Combining manual and automated testing

Choosing a cloud hosting partner with confidence

Test-driven development is only as good as its tests, and therein lies the problem for many developers. Recently, D Richard Hipp, who is the main author of the Sqlite open source database, analysed his source code. He calculates that 59 per cent of his code base is devoted to testing, covering 97.4 per cent of the code.

Creating comprehensive tests is hard work, and test-driven development represents a change in programming culture; it is not merely "another useful technique". Developers naturally look for a way to automate some of this work. This is inherently difficult. The only definitive description of an application's behaviour is in its code, yet the point of unit tests is that this code may be flawed. The ideal moment to create tests is just before the code is written, when the developer knows his precise intentions. Automation cannot do this for you. Manually written tests are also imperfect. Test authors may fail to cover all the possible scenarios, or may include bugs in the tests themselves. Since neither manual nor automated testing can cover all the bases, how about combining the two? Broadly speaking, this is the thinking behind Agitator, which describes itself as "an automated way of exercising your code without the necessity of writing tests first",

Screenshot shows Agitator running within the Eclipse IDE.

Agitator is implemented as an Eclipse plug-in, though it also integrates with IntelliJ IDEA and JBuilder. These IDE integrations simply take you to Eclipse, so it is Eclipse users that are most likely to enjoy Agitator.

The core feature of Agitator is that it "Agitates". In order to Agitate a class, the system first analyses the compiled byte-code for the purpose of generating tests. Since it works on compiled code, it is essential to Build before Agitating, or to use Eclipse's auto-build feature. Next, Agitator generates sample input data and calls the public members of the class, instantiating objects as needed. You had better make sure your application is not hooked up to a live database at this point, as Agitator will not be responsible for the results. Finally, the results are displayed as a list of observations and problems. You can set the Agitation level to Normal, Extended or Aggressive, so determining the number of times each method is called and the variety of input values.

So what is an observation? These are similar to assertions; statements about the running code which Agitator thinks might be interesting, such as that a certain variable or return value was never null, or fell in a range of values. Agitator also reports exceptions, failed assertions, and code coverage, showing how much of the code was exercised by the Agitation.

Screenshot showing Agitator’s observations as the starting point for tailoring an Agitation.

Most observations are inconsequential, but as you review these you may see unexpected results, perhaps exposing a bug. You can also add and edit the observations, improving on Agitator's guesses and, in effect, creating unit tests that build on the automated foundation. You will need to become familiar with Agitator's small Expression syntax, learning for example that @PRE means the value of an object before a method call, and @RETURN the return value of the method. Snapshots let you drill down into an observation to inspect the other conditions that generated the result. You will likely spot important observations that should be promoted to actual assertions, which you can do with a couple of clicks. This is Agitation at its most valuable, enabling you to identify the correct assertions to make about the behaviour of your code.

Unfortunately you will probably still find that the Agitation did not do quite what was wanted. Randomly generated values may be less useful than close simulations of the input your application will get in the real world. However, you can configure Agitator to supply these by using Agitator's Factories. Factories let you control how Agitator gets its test values, letting you specify anything from a specific range of a primitive type to custom code for constructing objects used by your application. You can also specify how often Agitator gets values from the custom Factory, as opposed to using its defaults. If all that sounds like too much work, you can automate factory configuration and other customisations by using Agitator's Experts. Experts are supplied for Hibernate, J2EE, Struts and Log4j, and you can modify these or create your own. Agitator can automatically create mock objects, too.

There is no need to abandon conventional unit tests in order to use Agitator. If JUnit tests are present, Agitator will call them in the same way as the JUnit harness, and show the outcome in its Agitation results panel. The two work well together [Ed: as you'd expect, since the people behind Agitator are also the people behind JUnit] and you can use Agitator to improve your JUnit tests. In some cases JUnit tests are an easier and more natural approach to test creation than Agitator.

Finally, Agitator has a code rules engine for static code analysis. A generous set of rules is supplied, including coding conventions, naming conventions, and metrics such as maximum number of statements in a method, J2EE standards and likely candidates for bugs. Some of the rules come from Oliver Burn's open source Checkstyle project.

Screenshot showing Agitator’s comprehensive static analysis rules.

A separate extra-cost product, the Agitar Management Dashboard, lets you easily analyse and monitor the health of a project in terms of test failures, coverage and adherence to code rules [Ed: I think this can be an important part of tying unit testing and development back to the project's business sponsors].

Agitator is an intelligent and well-crafted product, the outcome of deep thinking about how to automate unit testing. Static and code coverage analysis comes as part of the package. At just under £2000 per user it's not cheap, but in the right hands it will raise software quality, and provides an excellent framework for creating more effective tests. That said, it's emphatically not a quick-fix route to test-driven development. At worst it can generate large amounts of not very useful data to plough through, and it is not truly geared to test-first development (because you must have code to Agitate). Which is interesting, considering that Kent Beck is involved with Agitar; although you can do test-first development with Agitar and since testing and coding are iterative processes, arguably this may not matter much in practice. However, creating high quality Agitations takes work, and the product will be most useful to developers already skilled with JUnit. There's more information here.

Intelligent flash storage arrays

More from The Register

next story
UNIX greybeards threaten Debian fork over systemd plan
'Veteran Unix Admins' fear desktop emphasis is betraying open source
Netscape Navigator - the browser that started it all - turns 20
It was 20 years ago today, Marc Andreeesen taught the band to play
Redmond top man Satya Nadella: 'Microsoft LOVES Linux'
Open-source 'love' fairly runneth over at cloud event
Return of the Jedi – Apache reclaims web server crown
.london, .hamburg and .公司 - that's .com in Chinese - storm the web server charts
Chrome 38's new HTML tag support makes fatties FIT and SKINNIER
First browser to protect networks' bandwith using official spec
Admins! Never mind POODLE, there're NEW OpenSSL bugs to splat
Four new patches for open-source crypto libraries
prev story

Whitepapers

Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Cloud and hybrid-cloud data protection for VMware
Learn how quick and easy it is to configure backups and perform restores for VMware environments.
Three 1TB solid state scorchers up for grabs
Big SSDs can be expensive but think big and think free because you could be the lucky winner of one of three 1TB Samsung SSD 840 EVO drives that we’re giving away worth over £300 apiece.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.