Agitating Java and testing Windows

Test-driven development with Visual Studio Team System

It is interesting to contrast the work Agitar is doing with what Microsoft is delivering with Visual Studio Team System. JUnit has long been ported to .NET in the form of NUnit, but Team System does not use this open source product, preferring its own testing framework. Testing is a major focus of the product, influenced by two distinct (and somewhat contradictory) strands of thought. The first, and I think primary, influence is from Microsoft's own testing practice. There is a distinct tester role dedicated to finding and reporting bugs and monitoring software quality, identified in the process guidance expressed in the Microsoft Solutions Framework (MSF) and the Visual Studio 2005 Team Edition for Software Testers. The secondary influence comes from eXtreme Programming (XP), which teaches test-first development and argues that testing is an integral part of the developer's task, not something that should be delegated to others or postponed to a later stage. Even so, Team System does provide the necessary infrastructure for test-driven development, and may prove to be the beginning of a quiet revolution towards more disciplined and process-oriented development on Microsoft's platform.

The starting point is the unit testing built into both the Developer and Tester editions of Visual Studio. Test classes and methods are annotated with attributes, used by the test harness to run tests with configurable initialisation and cleanup. You can easily create unit tests using a wizard. A right-click in the code editor offers Create Unit Tests, which opens a dialog allowing you to select which methods to test. Visual Studio then generates skeleton tests for those methods. These are simple skeletons, not smart Agitations, though they are still great time-savers, especially for things like testing private members, which requires reflection, or testing web applications (available in the Tester edition only). By default, they fail with a result of Inconclusive; it is the developer's responsibility to implement the code that will provide meaningful results. You can easily make tests data-driven, iterating through all the values of a linked table to get input. You can also create suites of tests, and configure code coverage enabling you to see which code was exercised. It is all nicely integrated into Visual Studio, which has a test view where you can browse and run the available tests.

Those with the Tester edition or the full suite have additional tests available, including web tests. Load tests simulate multiple method calls or web requests over a set period, and are invaluable for testing performance, scalability and resilience. You can use multiple computers to do the testing, by setting up remote controllers and agents.

Team System has strong support for static code analysis, building on an earlier informal project called FxCop, with hundreds of rules you can apply through project properties.

One interesting feature of Team System, in terms of process, is the ability to set check-in policy, requiring certain standards to be met before it is possible to check-in code. It is possible for developers to override this, but doing so raises an administrator alert.

You can set three types of policy. One specifies code analysis rules, one specifies test policy, and the third requires associated work items. The test policy specifies one or more test lists that must succeed, although test lists can only be created with the Tester edition. A major omission in the first release is that you cannot set a rule to require that a certain proportion of code must be covered by tests before check-in - you can only specify pre-existing tests. Team System is highly extensible, so it's likely that Microsoft or third parties will plug this and other gaps in due course.

Team System is a huge step forward for Microsoft, bringing testing into the heart of the development process. That said, I would prefer to see less emphasis on the distinct Tester role and more emphasis on test-driven development. Some features in the Tester edition, like test management and web tests, should also be in the Developer version. This sort of product partitioning is always frustrating, especially for small teams or solo developers, who will still need the full suite.®

The licensing is so complex that Microsoft has a 15 page white paper (available here) to describe it, involving server and client licenses for Windows and Office as well as Team Foundation Server and Visual Studio. Basic prices are available here.

Still, it remains true that tool costs are generally only a small part of total development budgets, and Team System in its latest Release Candidate looks very worthwhile. More information can be found here.