Free ESG report : Seamless data management with Avere FXT

Comment I got some flack recently for daring to suggest (or appearing to) that open source software (OSS) should be "fit for purpose" (here).

After all, since all those saintly OSS developers are working for nothing, why should we expect their software to work? Well, I can't imagine a company with any hope of staying in business using software that isn't "fit for purpose", OSS or not. But, luckily, as I personally believe in OSS, there is objective evidence that it really can be pretty good quality.

This comes from the Java Open Review, an open source project sponsored by Fortify Software which uses Fortify SCA tools and Findbugs to look for defects in software – as a service. It publishes aggregated statistics but has a "responsible disclosure policy", which means details of bugs found are fed back only to the authors.

The project has recently analysed some common Java packages of the sort used to build/support other applications, including Hibernate, Struts, Spring, and Tomcat. These did pretty well, as you might expect, averaging under 0.1 defects per kloc (kilo lines of code), as opposed to the expected 20-30 defects per kloc reported by Carnegie Mellon's Cylab Sustainable Computing Consortium (although one should be a little cautious comparing such studies as even the definition of a "kloc" could differ).

Java, the most popular OSS language by far, appears to be more reliable than C/C++ - which is not exactly news, but it is always good to actually confirm what is obvious.

And it's useful ammunition for developers wanting to exploit OSS in conservative companies, as it appears that OSS may contain at least an order of magnitude fewer bugs than commercial software. Although the sample sizes are rather small as yet, and you can probably find buggy OSS if you look, I am pretty impressed not by the absolute figures so much as by the OSS community supporting an open assessment of OSS quality – this bodes well for OSS quality generally, if the project excites interest in the community. Perhaps we should revisit his project in a few months and look at interest levels.

Now, can you imagine Microsoft, IBM, or BEA publishing their defect statistics in any useful way? I can't, but if I'm wrong, please tell me. To be fair they'd all have to do so, I suppose, in some sort of race where no one wanted to be first.

But in the meantime, this from the Colorado State University makes interesting reading. It suggests that defect rates in open source operating systems are comfortably lower than those in Windows, although those in Windows really aren't too bad. Unfortunately, only a beta version of Windows XP was available, so you'd expect its defect rates to be higher – which perhaps ought to worry users of Web 2.0 applications where beta software sometimes seems like the norm.

You can find the Java Open Review study here (registration needed). Some other quality nuggets from this project are that cross-site scripting is the most common vulnerability you should be considering these days, and that even if Java packages are pretty good, the code samples supplied with them often don't reflect good security practice – just one reason why basically good OSS code is often used in insecure ways by developers.

Oh, and while I'm talking about security and finding defects, one of my pet hates is employing ex-hackers as penetration testers. Penetration testing has its place as a sort of acceptance testing or threat assessment, but it is really too late in the lifecycle to find defects anyway.

But, if you employ ex hackers to do it, how do you know they've really reformed (do you really want to give them low-level access to commercially sensitive or personal data in your systems), and how do you know they're as good as they say they are? Well, now there's an analysis tool from Fortify called Tracer which looks at the executables being penetration tested and reports back on coverage etc. That could sort out the sheep from the goats! ®

5 ways to reduce advertising network latency