Feeds

Fortify and the Java open review project

Is open source software better than you think?

Boost IT visibility and business value

Comment I got some flack recently for daring to suggest (or appearing to) that open source software (OSS) should be "fit for purpose" (here).

After all, since all those saintly OSS developers are working for nothing, why should we expect their software to work? Well, I can't imagine a company with any hope of staying in business using software that isn't "fit for purpose", OSS or not. But, luckily, as I personally believe in OSS, there is objective evidence that it really can be pretty good quality.

This comes from the Java Open Review, an open source project sponsored by Fortify Software which uses Fortify SCA tools and Findbugs to look for defects in software – as a service. It publishes aggregated statistics but has a "responsible disclosure policy", which means details of bugs found are fed back only to the authors.

The project has recently analysed some common Java packages of the sort used to build/support other applications, including Hibernate, Struts, Spring, and Tomcat. These did pretty well, as you might expect, averaging under 0.1 defects per kloc (kilo lines of code), as opposed to the expected 20-30 defects per kloc reported by Carnegie Mellon's Cylab Sustainable Computing Consortium (although one should be a little cautious comparing such studies as even the definition of a "kloc" could differ).

Java, the most popular OSS language by far, appears to be more reliable than C/C++ - which is not exactly news, but it is always good to actually confirm what is obvious.

And it's useful ammunition for developers wanting to exploit OSS in conservative companies, as it appears that OSS may contain at least an order of magnitude fewer bugs than commercial software. Although the sample sizes are rather small as yet, and you can probably find buggy OSS if you look, I am pretty impressed not by the absolute figures so much as by the OSS community supporting an open assessment of OSS quality – this bodes well for OSS quality generally, if the project excites interest in the community. Perhaps we should revisit his project in a few months and look at interest levels.

Now, can you imagine Microsoft, IBM, or BEA publishing their defect statistics in any useful way? I can't, but if I'm wrong, please tell me. To be fair they'd all have to do so, I suppose, in some sort of race where no one wanted to be first.

But in the meantime, this from the Colorado State University makes interesting reading. It suggests that defect rates in open source operating systems are comfortably lower than those in Windows, although those in Windows really aren't too bad. Unfortunately, only a beta version of Windows XP was available, so you'd expect its defect rates to be higher – which perhaps ought to worry users of Web 2.0 applications where beta software sometimes seems like the norm.

You can find the Java Open Review study here (registration needed). Some other quality nuggets from this project are that cross-site scripting is the most common vulnerability you should be considering these days, and that even if Java packages are pretty good, the code samples supplied with them often don't reflect good security practice – just one reason why basically good OSS code is often used in insecure ways by developers.

Oh, and while I'm talking about security and finding defects, one of my pet hates is employing ex-hackers as penetration testers. Penetration testing has its place as a sort of acceptance testing or threat assessment, but it is really too late in the lifecycle to find defects anyway.

But, if you employ ex hackers to do it, how do you know they've really reformed (do you really want to give them low-level access to commercially sensitive or personal data in your systems), and how do you know they're as good as they say they are? Well, now there's an analysis tool from Fortify called Tracer which looks at the executables being penetration tested and reports back on coverage etc. That could sort out the sheep from the goats! ®

The essential guide to IT transformation

More from The Register

next story
Intel's Raspberry Pi rival Galileo can now run Windows
Behold the Internet of Things. Wintel Things
The Return of BSOD: Does ANYONE trust Microsoft patches?
Sysadmins, you're either fighting fires or seen as incompetents now
Munich considers dumping Linux for ... GULP ... Windows!
Give a penguinista a hug, the Outlook's not good for open source's poster child
Microsoft cries UNINSTALL in the wake of Blue Screens of Death™
Cache crash causes contained choloric calamity
Eat up Martha! Microsoft slings handwriting recog into OneNote on Android
Freehand input on non-Windows kit for the first time
Linux kernel devs made to finger their dongles before contributing code
Two-factor auth enabled for Kernel.org repositories
prev story

Whitepapers

5 things you didn’t know about cloud backup
IT departments are embracing cloud backup, but there’s a lot you need to know before choosing a service provider. Learn all the critical things you need to know.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Build a business case: developing custom apps
Learn how to maximize the value of custom applications by accelerating and simplifying their development.
Rethinking backup and recovery in the modern data center
Combining intelligence, operational analytics, and automation to enable efficient, data-driven IT organizations using the HP ABR approach.
Next gen security for virtualised datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.