Agitating Java and testing Windows
Combining manual and automated testing
Test-driven development is only as good as its tests, and therein lies the problem for many developers. Recently, D Richard Hipp, who is the main author of the Sqlite open source database, analysed his source code. He calculates that 59 per cent of his code base is devoted to testing, covering 97.4 per cent of the code.
Creating comprehensive tests is hard work, and test-driven development represents a change in programming culture; it is not merely "another useful technique". Developers naturally look for a way to automate some of this work. This is inherently difficult. The only definitive description of an application's behaviour is in its code, yet the point of unit tests is that this code may be flawed. The ideal moment to create tests is just before the code is written, when the developer knows his precise intentions. Automation cannot do this for you. Manually written tests are also imperfect. Test authors may fail to cover all the possible scenarios, or may include bugs in the tests themselves. Since neither manual nor automated testing can cover all the bases, how about combining the two? Broadly speaking, this is the thinking behind Agitator, which describes itself as "an automated way of exercising your code without the necessity of writing tests first",
Agitator is implemented as an Eclipse plug-in, though it also integrates with IntelliJ IDEA and JBuilder. These IDE integrations simply take you to Eclipse, so it is Eclipse users that are most likely to enjoy Agitator.
The core feature of Agitator is that it "Agitates". In order to Agitate a class, the system first analyses the compiled byte-code for the purpose of generating tests. Since it works on compiled code, it is essential to Build before Agitating, or to use Eclipse's auto-build feature. Next, Agitator generates sample input data and calls the public members of the class, instantiating objects as needed. You had better make sure your application is not hooked up to a live database at this point, as Agitator will not be responsible for the results. Finally, the results are displayed as a list of observations and problems. You can set the Agitation level to Normal, Extended or Aggressive, so determining the number of times each method is called and the variety of input values.
So what is an observation? These are similar to assertions; statements about the running code which Agitator thinks might be interesting, such as that a certain variable or return value was never null, or fell in a range of values. Agitator also reports exceptions, failed assertions, and code coverage, showing how much of the code was exercised by the Agitation.
Most observations are inconsequential, but as you review these you may see unexpected results, perhaps exposing a bug. You can also add and edit the observations, improving on Agitator's guesses and, in effect, creating unit tests that build on the automated foundation. You will need to become familiar with Agitator's small Expression syntax, learning for example that @PRE means the value of an object before a method call, and @RETURN the return value of the method. Snapshots let you drill down into an observation to inspect the other conditions that generated the result. You will likely spot important observations that should be promoted to actual assertions, which you can do with a couple of clicks. This is Agitation at its most valuable, enabling you to identify the correct assertions to make about the behaviour of your code.
Unfortunately you will probably still find that the Agitation did not do quite what was wanted. Randomly generated values may be less useful than close simulations of the input your application will get in the real world. However, you can configure Agitator to supply these by using Agitator's Factories. Factories let you control how Agitator gets its test values, letting you specify anything from a specific range of a primitive type to custom code for constructing objects used by your application. You can also specify how often Agitator gets values from the custom Factory, as opposed to using its defaults. If all that sounds like too much work, you can automate factory configuration and other customisations by using Agitator's Experts. Experts are supplied for Hibernate, J2EE, Struts and Log4j, and you can modify these or create your own. Agitator can automatically create mock objects, too.
There is no need to abandon conventional unit tests in order to use Agitator. If JUnit tests are present, Agitator will call them in the same way as the JUnit harness, and show the outcome in its Agitation results panel. The two work well together [Ed: as you'd expect, since the people behind Agitator are also the people behind JUnit] and you can use Agitator to improve your JUnit tests. In some cases JUnit tests are an easier and more natural approach to test creation than Agitator.
Finally, Agitator has a code rules engine for static code analysis. A generous set of rules is supplied, including coding conventions, naming conventions, and metrics such as maximum number of statements in a method, J2EE standards and likely candidates for bugs. Some of the rules come from Oliver Burn's open source Checkstyle project.
A separate extra-cost product, the Agitar Management Dashboard, lets you easily analyse and monitor the health of a project in terms of test failures, coverage and adherence to code rules [Ed: I think this can be an important part of tying unit testing and development back to the project's business sponsors].
Agitator is an intelligent and well-crafted product, the outcome of deep thinking about how to automate unit testing. Static and code coverage analysis comes as part of the package. At just under £2000 per user it's not cheap, but in the right hands it will raise software quality, and provides an excellent framework for creating more effective tests. That said, it's emphatically not a quick-fix route to test-driven development. At worst it can generate large amounts of not very useful data to plough through, and it is not truly geared to test-first development (because you must have code to Agitate). Which is interesting, considering that Kent Beck is involved with Agitar; although you can do test-first development with Agitar and since testing and coding are iterative processes, arguably this may not matter much in practice. However, creating high quality Agitations takes work, and the product will be most useful to developers already skilled with JUnit. There's more information here.