Agitating Java and testing Windows

Combining manual and automated testing

Securing Web Applications Made Simple and Scalable

Test-driven development is only as good as its tests, and therein lies the problem for many developers. Recently, D Richard Hipp, who is the main author of the Sqlite open source database, analysed his source code. He calculates that 59 per cent of his code base is devoted to testing, covering 97.4 per cent of the code.

Creating comprehensive tests is hard work, and test-driven development represents a change in programming culture; it is not merely "another useful technique". Developers naturally look for a way to automate some of this work. This is inherently difficult. The only definitive description of an application's behaviour is in its code, yet the point of unit tests is that this code may be flawed. The ideal moment to create tests is just before the code is written, when the developer knows his precise intentions. Automation cannot do this for you. Manually written tests are also imperfect. Test authors may fail to cover all the possible scenarios, or may include bugs in the tests themselves. Since neither manual nor automated testing can cover all the bases, how about combining the two? Broadly speaking, this is the thinking behind Agitator, which describes itself as "an automated way of exercising your code without the necessity of writing tests first",

Screenshot shows Agitator running within the Eclipse IDE.

Agitator is implemented as an Eclipse plug-in, though it also integrates with IntelliJ IDEA and JBuilder. These IDE integrations simply take you to Eclipse, so it is Eclipse users that are most likely to enjoy Agitator.

The core feature of Agitator is that it "Agitates". In order to Agitate a class, the system first analyses the compiled byte-code for the purpose of generating tests. Since it works on compiled code, it is essential to Build before Agitating, or to use Eclipse's auto-build feature. Next, Agitator generates sample input data and calls the public members of the class, instantiating objects as needed. You had better make sure your application is not hooked up to a live database at this point, as Agitator will not be responsible for the results. Finally, the results are displayed as a list of observations and problems. You can set the Agitation level to Normal, Extended or Aggressive, so determining the number of times each method is called and the variety of input values.

So what is an observation? These are similar to assertions; statements about the running code which Agitator thinks might be interesting, such as that a certain variable or return value was never null, or fell in a range of values. Agitator also reports exceptions, failed assertions, and code coverage, showing how much of the code was exercised by the Agitation.

Screenshot showing Agitator’s observations as the starting point for tailoring an Agitation.

Most observations are inconsequential, but as you review these you may see unexpected results, perhaps exposing a bug. You can also add and edit the observations, improving on Agitator's guesses and, in effect, creating unit tests that build on the automated foundation. You will need to become familiar with Agitator's small Expression syntax, learning for example that @PRE means the value of an object before a method call, and @RETURN the return value of the method. Snapshots let you drill down into an observation to inspect the other conditions that generated the result. You will likely spot important observations that should be promoted to actual assertions, which you can do with a couple of clicks. This is Agitation at its most valuable, enabling you to identify the correct assertions to make about the behaviour of your code.

Unfortunately you will probably still find that the Agitation did not do quite what was wanted. Randomly generated values may be less useful than close simulations of the input your application will get in the real world. However, you can configure Agitator to supply these by using Agitator's Factories. Factories let you control how Agitator gets its test values, letting you specify anything from a specific range of a primitive type to custom code for constructing objects used by your application. You can also specify how often Agitator gets values from the custom Factory, as opposed to using its defaults. If all that sounds like too much work, you can automate factory configuration and other customisations by using Agitator's Experts. Experts are supplied for Hibernate, J2EE, Struts and Log4j, and you can modify these or create your own. Agitator can automatically create mock objects, too.

There is no need to abandon conventional unit tests in order to use Agitator. If JUnit tests are present, Agitator will call them in the same way as the JUnit harness, and show the outcome in its Agitation results panel. The two work well together [Ed: as you'd expect, since the people behind Agitator are also the people behind JUnit] and you can use Agitator to improve your JUnit tests. In some cases JUnit tests are an easier and more natural approach to test creation than Agitator.

Finally, Agitator has a code rules engine for static code analysis. A generous set of rules is supplied, including coding conventions, naming conventions, and metrics such as maximum number of statements in a method, J2EE standards and likely candidates for bugs. Some of the rules come from Oliver Burn's open source Checkstyle project.

Screenshot showing Agitator’s comprehensive static analysis rules.

A separate extra-cost product, the Agitar Management Dashboard, lets you easily analyse and monitor the health of a project in terms of test failures, coverage and adherence to code rules [Ed: I think this can be an important part of tying unit testing and development back to the project's business sponsors].

Agitator is an intelligent and well-crafted product, the outcome of deep thinking about how to automate unit testing. Static and code coverage analysis comes as part of the package. At just under £2000 per user it's not cheap, but in the right hands it will raise software quality, and provides an excellent framework for creating more effective tests. That said, it's emphatically not a quick-fix route to test-driven development. At worst it can generate large amounts of not very useful data to plough through, and it is not truly geared to test-first development (because you must have code to Agitate). Which is interesting, considering that Kent Beck is involved with Agitar; although you can do test-first development with Agitar and since testing and coding are iterative processes, arguably this may not matter much in practice. However, creating high quality Agitations takes work, and the product will be most useful to developers already skilled with JUnit. There's more information here.

Bridging the IT gap between rising business demands and ageing tools

More from The Register

next story
NO MORE ALL CAPS and other pleasures of Visual Studio 14
Unpicking a packed preview that breaks down ASP.NET
Cheer up, Nokia fans. It can start making mobes again in 18 months
The real winner of the Nokia sale is *drumroll* ... Nokia
Mozilla fixes CRITICAL security holes in Firefox, urges v31 upgrade
Misc memory hazards 'could be exploited' - and guess what, one's a Javascript vuln
Put down that Oracle database patch: It could cost $23,000 per CPU
On-by-default INMEMORY tech a boon for developers ... as long as they can afford it
Google shows off new Chrome OS look
Athena springs full-grown from Chromium project's head
Apple: We'll unleash OS X Yosemite beta on the MASSES on 24 July
Starting today, regular fanbois will be guinea pigs, it tells Reg
HIDDEN packet sniffer spy tech in MILLIONS of iPhones, iPads – expert
Don't panic though – Apple's backdoor is not wide open to all, guru tells us
prev story


Designing a Defense for Mobile Applications
Learn about the various considerations for defending mobile applications - from the application architecture itself to the myriad testing technologies.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Top 8 considerations to enable and simplify mobility
In this whitepaper learn how to successfully add mobile capabilities simply and cost effectively.
Seven Steps to Software Security
Seven practical steps you can begin to take today to secure your applications and prevent the damages a successful cyber-attack can cause.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.