Feeds

Crowdsourced flaw-finding cheaper than in-house bug hunters

Study shows bug bounties for browsers beats hiring humans

Providing a secure and efficient Helpdesk

A study into the once-controversial practice of vulnerability rewards programs (VRPs) – paying researchers bug bounties for reporting security flaws – has found that for browser builders, the practice is not only more effective at spotting problems that hiring code-checkers, it's also much better value for the money.

"We find that VRPs appear to provide an economically efficient mechanism for finding vulnerabilities, with a reasonable cost/benefit trade-off, "the paper from the University of California, Berkeley computer science department states. "In particular, they appear to be 2-100 times more cost effective than hiring expert security researchers to find vulnerabilities."

The paper, "An Empirical Study of Vulnerability Rewards Programs" by Matthew Finifter, Devdatta Akhawe, and David Wagner, uses three years of data from two vulnerability reward programs run by Google's Chrome and Mozilla's Firefox. Google paid out $579,605 in bounties during that period; Mozilla shelled out $570,000.

This translates to between $485 and $658 a day for the two companies, while "an average North American developer" costs around $500 per day, assuming a $100,000 salary with a 50 per cent overhead for costs such as healthcare and office space.

Over the three-year period, Google paid bounties for outsiders who spotted 371 Chrome flaws, well outperforming its best internal security researcher who got 263. With Firefox, the crowd found 148 flaws, compared to just 48 for the best internal team member, although Mozilla doesn't have anything like the hiring budget of the Chocolate Factory.

In addition, the range of flaws the bounty programs produced is much more broad because of the range of people trying different techniques to find them. While this study concentrates on browsers, the researchers suggest the model works for the rest of the software industry.

"The cost/benefit trade-off may vary for other types of (i.e., non-browser) software vendors; in particular, the less costly a security incident is for a vendor, the less useful we can expect a VRP to be," the study states. "Additionally, we expect that the higher-profile the software project is (among developers and security researchers), the more effective a VRP will be."

The Lottery Effect

Google decided to start a VRP program in January 2010, paying a fee of $500 for serious security holes and a special bonus $1337 reward for critical or clever flaw discoveries. But Google also offers big bucks at its Pwnium hacking contests, with $3.14159m up for grabs at the most recent hackathon.

Firefox takes a very different approach. It was one of the first to adopt a formal VRP program in 2004, copying Netscape's example in 1995, and initially paid $500 per serious flaw before increasing the fee to $3,000 in 2010 after Google came on the scene. It's a set fee for serious flaws.

Of the two systems, the researchers found Google's approach works best, thanks in part to the Lottery Effect. Just over 84 per cent of Google's bounty payments are $1,000 or less, but the big money events lure researchers into the field with a few very big cash prizes, and so the Chocolate Factory gets a lot more flaws reported than Firefox.

But Google gets good value for its big prizes as well, the study found. One Pwnium contest winner uncovered a flaw so serious that Google conducted a full review of the Chrome kernel file API and found a rat's nest of other vulnerabilities stemming from the same issue.

Based on the study data, being an independent bug-rustler isn't a career that's going to pay the rent on its own. One Firefox researcher earned $141,000 over the three-year period and three Chrome flaw-finders made $80,000 apiece, but only six and five people respectively made $20,000 or more.

"Contributing to a single VRP is, in general, not a viable full-time job, though contributing to multiple VRPs may be, especially for unusually successful vulnerability researchers," the team hypothesizes.

However, it may be a route to full-time employment for those looking to get into the field. Successful flaw finders get noticed and hired by firms that specialize in the field, or by the browser manufacturers themselves, the team suggests.

Meanwhile El Reg suggests that Apple and Microsoft take a good long look at the study. Microsoft has recently broken its bug-bounty virginity with a $150,000 hacking contest at this year's Black Hat security conference, but there's no sign that Apple's likely to budge on its longstanding policy of not paying for bugs. ®

Choosing a cloud hosting partner with confidence

More from The Register

next story
SMASH the Bash bug! Apple and Red Hat scramble for patch batches
'Applying multiple security updates is extremely difficult'
Shellshock: 'Larger scale attack' on its way, warn securo-bods
Not just web servers under threat - though TENS of THOUSANDS have been hit
Apple's new iPhone 6 vulnerable to last year's TouchID fingerprint hack
But unsophisticated thieves need not attempt this trick
Hackers thrash Bash Shellshock bug: World races to cover hole
Update your gear now to avoid early attacks hitting the web
Oracle SHELLSHOCKER - data titan lists unpatchables
Database kingpin lists 32 products that can't be patched (yet) as GNU fixes second vuln
Who.is does the Harlem Shake
Blame it on LOLing XSS terroristas
Researchers tell black hats: 'YOU'RE SOOO PREDICTABLE'
Want to register that domain? We're way ahead of you.
Stunned by Shellshock Bash bug? Patch all you can – or be punished
UK data watchdog rolls up its sleeves, polishes truncheon
prev story

Whitepapers

A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.
Storage capacity and performance optimization at Mizuno USA
Mizuno USA turn to Tegile storage technology to solve both their SAN and backup issues.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Beginner's guide to SSL certificates
De-mystify the technology involved and give you the information you need to make the best decision when considering your online security options.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.