Open-source projects get free checkup by automated tools

More open-source software projects are gaining the benefits of the latest code-checking software, as the programs' makers look to prove their worth.

On Tuesday, code-analysis software maker Coverity announced that its automated bug finding tool had analyzed the community-built operating system FreeBSD and flagged 306 potential software flaws, or about one issue for every 4,000 lines of code. The tool, which identifies certain types of programming errors, has previously been used to find flaws in other open-source software, including the Linux kernel and the MySQL database.

The low number of flaws found by the system underscores that FreeBSD's manual auditing by project members has reduced the vulnerabilities in the operating system, said Seth Hallem, CEO of Coverity.

"FreeBSD - as well as OpenBSD and NetBS - are small communities which have made it a priority to build security into the operating system, and that has paid dividends," Hallem said.

FreeBSD is the latest open-source project to benefit from being run through its paces by code-checking software. Last year, Coverity's tools found 950 potential flaws in version 2.6.9 of the Linux kernel, 97 potential flaws in the MySQL database code, and 26 potential flaws in the Berkeley DB code. Moreover, the tool has analyzed the code for OpenBSD, with any flaws found submitted back to the project, according to OpenBSD members.

"The open-source world as gone on a huge bug hunt for low-hanging fruit in the major packages," said Adam Shostack, chief technology officer for code-analysis tool maker Reflective. "Commercial organizations with closed source have not, and our customers often find things that surprise them."

Reflective has "pretty cool" plans for open-source auditing, Shostack said. He did not elaborate, however.

Since 2002, about 4,000 vulnerabilities have been found and identified annually by security researchers, companies, and hackers, according to the statistics from the Computer Emergency Response Team (CERT) Coordination Center. Fixing such flaws after product development is expensive: The cost to identify and patch vulnerabilities in the United States' software infrastructure costs anywhere from $22 billion to $60 billion annually, according to estimates by the National Institute of Standards and Technology.

To avoid paying the cost to fix bugs after a product ships, companies are increasingly using automated tools to audit their code. Coverity counts database maker Oracle and graphics chip maker Nvidia among its customers. Analysis tools created by Fortify Software have been used by AT&T Wireless and online payment service Paypal, a subsidiary of online auctioneer eBay. And, Agitar's Java analysis tool has audited software for trading service MarketAxess and portfolio management service Financial Engines. Reflective has not released the names of its customers.

Storage system provider Veritas Software is also a customer of Coverity. Symantec, which owns SecurityFocus, plans to acquire Veritas.

Microsoft has also added code-analysis tools as a major part of the software giant's revamp of its application development process. In 1999, the company bought Intrinsa, a maker of bug-finding software, for $60 million. Microsoft now requires that all software be run through its PREfast code checker on a daily basis and the more comprehensive PREfix analysis tool for significant builds.

While code-checking tools do not find all the flaws in software, the programs are very good at finding certain classes of software problems, said Theo de Raadt, project leader for OpenBSD.

"Most bugs in software are the same ten to fifteen mistakes made over and over," he said. "Automated checkers can find certain classes of these bugs quite easily. All bugs of this kind are worth fixing, but very, very few people are fixing them or are even aware of how simple these things are."

Not all the potential flaws found by analysis tools are security holes. For FreeBSD, while 306 problems were flagged by Coverity's software, only 5 issues could be triggered by user input. The software classified another 12 vulnerabilities as buffer overruns, another potentially serious security issue.

The FreeBSD project has analyzed the flaws and fixed the issues, said Colin Percival, visiting researcher at Simon Fraser University and the deputy security officer for the FreeBSD project.

"Anyone who is reporting bugs, we will fix them," he said, adding that code checkers are a way to insure that developers do not make easily detectable mistakes. "Having these automated checks for all standards security flaws is the way things are going."

While testing analysis tools on open-source projects has helped companies improve their products, community software has also reaped the security benefits.

For instance, FreeBSD has doubled in size in the past year and Coverity has added improvements to its tool, but the company only found half as many bugs as a year ago. Similarly, while the Linux source code tripled in size - including driver software - since the Linux kernel 2.4.1 was released in 2001, Coverity's tools only flagged half as many flaws in the latest audit compared to four years ago. Moreover, more than half of all flaws in Linux occured in the device drivers, and only one per cent of the errors were found in the core kernel code.

"These tools can be viewed as a super member of the community that does a good job of finding and reporting bugs," said Bill Weinberg, open-source architecture specialist for the Open Source Development Labs, the non-profit organization that helps manage Linux development.

In the search for consistently high-quality code, such tools have become important, Weinberg added.

"The are a set of incredibly acute eyeballs looking at the code in ways that programmers, white hats and even black hats can't do," he said.

Copyright © 2005,

Related stories

Security Report: Windows vs Linux
MS mulls external testing for security patches
Too cool for secure code