Crypto hash boffins trip on buffer overflow

Two of the programs submitted in the first round of a competition to find the next cryptographic hash standard contain buffer overflow errors that could make them prone to crashes and security problems.

The discovery, by researchers at Fortify Software, just goes to show that even the world's most foremost security geeks can slip up when implementing their latest and greatest creations. In one case, a team lead by Massachusetts Institute of Technology professor Ronald Rivest - the R in the RSA cryptography algorithm - submitted a reference implementation that contained three buffer overflows. A second entry, dubbed Blender and submitted by Colin Bradbury, contained one.

Like all the contest submissions, the buggy reference implementations were written in C, often considered the Corvair of computer languages because of its susceptibility to critical security bugs.

"This just emphasizes what we already knew about C," Fortify researchers wrote. Even the most careful, security conscious developer messes up memory management."

The implementation submitted by Rivest's team suffered from a hashval field with a buffer that is read or written using different bounds. It's unlikely the mistakes would result in problems when the program is built using the standard ANSI compiler, but other compilers could allow it to be exploited. The bug was fixed by doubling the size of the vulnerable buffer, Fortify said.

The buffer overflow in Blender was the result of a typo. A developer mistyped3 instead of 2 for an array access. It required very large input in order to be caught.

The submissions are part of the competition sponsored by the National Institute of Standards and Technology, which is seeking a replacement to the current hash standards such as MD4, MD5, and SHA-0. Over the past few years, a growing body of research has revealed those algorithms can be vulnerable to so-called "collisions," in which two separate data files generate the same digital signature.

Fortify's analysis has already been greeted by a chorus of critics who point out the errors in the submission lie with source-code implementations rather than the soundness of the underlying algorithm.

Fortify doesn't deny this, but it does argue that reference implementations have a very long shelf life and that this means their weaknesses are likely to be with us for a long time.

"Reference implementations don't disappear, they serve as a starting point for future implementations or are used directly," the company stated. "A bug in the RSA reference implementation was responsible for vulnerabilities in OpenSSL and to separate SSH implementations. They can also be used to design hardware implementations, using buffer sizes to decide how much silicon should be used." ®