Feeds

Of automated test case management

An interview with Keith Vanden Eynden

Top three mobile application threats

We recently published an article on all-pairs testing (read it here) by Keith Vanden Eynden of Seapine Software, in which he was enjoined not to “push” Seapine’s tools.

Nevertheless, Seapine’s TestTrack TCM tool is still an interesting product, accessible to the smaller business, so we asked Keith about using it in practice – and, especially, about using it to implement All-Pairs Testing.

David Norfolk, for Reg Dev: OK, very basic, what does TCM stand for?

Vanden Eynden:TestTrack TCM is our Test Case Management solution.

Reg Dev: And what aspects of testing does it cover?

Vanden Eynden:TestTrack TCM handles all of the tasks associated with the software testing process including test case creation, test scheduling and execution, plus test results measurement and reporting.

Reg Dev: Surely, all test case management tools worthy of the name do all that – or am I being naive? What makes TestTrack TCM special?

Vanden Eynden: Other test case management tools don’t offer as comprehensive a set of functions and aren’t as easy to install, customize and use. The biggest differentiator [according to Vanden Eynden], however, is TestTrack TCM’s ability to push test failures immediately and directly into your defect management workflow. This saves time by eliminating manual data entry and the errors it inevitably introduces. In addition, with this connection between test case management and defect management, software engineers are provided with the steps needed to reproduce a bug. This also saves them time and increases the probability of a first time fix of the issue.

TestTrack TCM also supports automated testing tools, such as QA Wizard Pro, so it can automate everything from building the test runs, running the automated tests, to moving any resulting defects into the defect management workflow.

Reg Dev: OK, given that we’re using Test Track for now, how can we use it to demonstrate All-Pairs Testing?

Vanden Eynden:TestTrack TCM’s test variants allow you to define the variables that apply to the applications you are testing and also allow you to implement all-pairs testing easily. The sample project installed with TestTrack TCM contains the variants and values shown in Figure 1.

Figure 1: TestTrack sample variants.

First you have to design and construct your all-pairs test runs. Then you begin by customizing the Database and Client Type variant values and adding a new Browser variant – see Figure 2.

Figure 2 Extended variant table.

With these variants in place, you can create one test case for the single sign-on feature and then generate test runs to cover different all-pairs combinations.

Reg Dev: OK then, how do you create the test case and run the tests?

Vanden Eynden:You can create a single test case and use different variant values to generate a range of test runs and provide the desired test coverage. For example, you create one test case for the single sign-on feature that contains all the information about how to set up and run the single sign-on test – see Figure 3.

Figure 3: Completed Add Test Case window.

Then you attach the all-pairs matrix to the test case to specify which variant value combinations to use when generating test runs – see Figure 4

Figure 4: All-pairs matrix attached to the test case.

After the test case is reviewed and approved, you can generate the 12 test runs identified earlier. You generate the test runs individually using the all-pairs matrix to complete the Generate Test Runs dialog box – see Figures 5 and 6.

Figure 5: the All Pairs test matrix.

Figure 6: The Generate Test Runs dialog box for the first test run.

After the first test run is generated, repeat the process to generate all the required combinations – see Figure 7.

Figure 7: Test runs generated for all the pairs of variant values.

Reg Dev: Just one point I’m not clear about. Where/when where does the actual test case code get written or generated? Does the QA team have to at least write the tests once? And then is it possible for test code to be stored in a `library’ and called as required?

Vanden Eynden: Of course someone has to author each test case, but, once written, they can be stored and used as templates for another project, or modified as needed to create new test cases.

Reg Dev: Well, very impressive, but what are the advantages of this approach?

Vanden Eynden: All-pairs testing enables you to achieve suitable test coverage without testing every possible combination. All-pairs testing can also reduce the number of test runs by 75 percent or more. With that time saving, you will be able to spend more time developing complex tests and less time performing mundane tests on the combination of variables.

Reg Dev: And, are there any disadvantages?

Vanden Eynden: Since you are not testing every possible combination, you introduce a little risk into the testing process, but it is a managed risk. One has to realize that all-pairs testing is a cost-benefit compromise, but a reasonable trade-off for most situations.

Reg Dev: Will you still need other approaches to testing too? What are they?

Vanden Eynden: No software testing technique can be expected to find all bugs. If all-pairs testing is an appropriate choice for your project, use it in combination with unit testing, code reviews, exploratory testing and other applicable tools that your organization regularly uses.

Reg Dev: What about managing testing as a whole?

Vanden Eynden: First, you have to invest in proper test planning and the creation of test cases. This is a big hurdle for many companies, but a necessary step to really getting control over quality. While it is hard to imagine developing software without defect tracking and source control tools in your arsenal, it wasn’t so long ago that these tools were seen as a luxury or non-essential. Tools for managing the testing process as a whole are still in that early stage. Products like TestTrack TCM make the management of test cases much easier and help companies see where they are light on test coverage. Once you have a tool like this, which saves you time and improves your ability to measure progress and quality, it is hard to go back to spreadsheets, Word documents, or paper.

Reg Dev: What about linking the things you’re are testing back to the Business Requirements which are, presumably, behind the application functions you are testing?

Vanden Eynden: This is desirable in many instances and required in some. TestTrack Pro can manage the business requirements and, as we discussed earlier, the integration between TestTrack TCM and our other tools provides this link between test outcomes and the managed requirements.

Reg Dev: Finally, how do you see Seapine’s tools developing in the face of evolving ALM (Application Lifecycle Management) offerings and the rise of Eclipse?

Vanden Eynden: Major releases of our flagship products are now entering the quality assurance phase and we have several new products on the drawing board. We see ourselves as part of the evolution of ALM so we think we are on the right track with our product set. Eclipse tools have their place and their supporters but there’s a need for commercial tools as well. Our goal is to offer a commercial ALM platform that is accessible from Eclipse wherever possible.

Reg Dev: Thanks. ®

Maximizing your infrastructure through virtualization

More from The Register

next story
HIDDEN packet sniffer spy tech in MILLIONS of iPhones, iPads – expert
Don't panic though – Apple's backdoor is not wide open to all, guru tells us
NO MORE ALL CAPS and other pleasures of Visual Studio 14
Unpicking a packed preview that breaks down ASP.NET
Captain Kirk sets phaser to SLAUGHTER after trying new Facebook app
William Shatner less-than-impressed by Zuck's celebrity-only app
Mozilla fixes CRITICAL security holes in Firefox, urges v31 upgrade
Misc memory hazards 'could be exploited' - and guess what, one's a Javascript vuln
Apple fanbois SCREAM as update BRICKS their Macbook Airs
Ragegasm spills over as firmware upgrade kills machines
Cheer up, Nokia fans. It can start making mobes again in 18 months
The real winner of the Nokia sale is *drumroll* ... Nokia
EU dons gloves, pokes Google's deals with Android mobe makers
El Reg cops a squint at investigatory letters
Chrome browser has been DRAINING PC batteries for YEARS
Google is only now fixing ancient, energy-sapping bug
prev story

Whitepapers

Top three mobile application threats
Prevent sensitive data leakage over insecure channels or stolen mobile devices.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Top 8 considerations to enable and simplify mobility
In this whitepaper learn how to successfully add mobile capabilities simply and cost effectively.
Application security programs and practises
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
The Essential Guide to IT Transformation
ServiceNow discusses three IT transformations that can help CIO's automate IT services to transform IT and the enterprise.