Feeds

Crowdsourced flaw-finding cheaper than in-house bug hunters

Study shows bug bounties for browsers beats hiring humans

The Power of One eBook: Top reasons to choose HP BladeSystem

A study into the once-controversial practice of vulnerability rewards programs (VRPs) – paying researchers bug bounties for reporting security flaws – has found that for browser builders, the practice is not only more effective at spotting problems that hiring code-checkers, it's also much better value for the money.

"We find that VRPs appear to provide an economically efficient mechanism for finding vulnerabilities, with a reasonable cost/benefit trade-off, "the paper from the University of California, Berkeley computer science department states. "In particular, they appear to be 2-100 times more cost effective than hiring expert security researchers to find vulnerabilities."

The paper, "An Empirical Study of Vulnerability Rewards Programs" by Matthew Finifter, Devdatta Akhawe, and David Wagner, uses three years of data from two vulnerability reward programs run by Google's Chrome and Mozilla's Firefox. Google paid out $579,605 in bounties during that period; Mozilla shelled out $570,000.

This translates to between $485 and $658 a day for the two companies, while "an average North American developer" costs around $500 per day, assuming a $100,000 salary with a 50 per cent overhead for costs such as healthcare and office space.

Over the three-year period, Google paid bounties for outsiders who spotted 371 Chrome flaws, well outperforming its best internal security researcher who got 263. With Firefox, the crowd found 148 flaws, compared to just 48 for the best internal team member, although Mozilla doesn't have anything like the hiring budget of the Chocolate Factory.

In addition, the range of flaws the bounty programs produced is much more broad because of the range of people trying different techniques to find them. While this study concentrates on browsers, the researchers suggest the model works for the rest of the software industry.

"The cost/benefit trade-off may vary for other types of (i.e., non-browser) software vendors; in particular, the less costly a security incident is for a vendor, the less useful we can expect a VRP to be," the study states. "Additionally, we expect that the higher-profile the software project is (among developers and security researchers), the more effective a VRP will be."

The Lottery Effect

Google decided to start a VRP program in January 2010, paying a fee of $500 for serious security holes and a special bonus $1337 reward for critical or clever flaw discoveries. But Google also offers big bucks at its Pwnium hacking contests, with $3.14159m up for grabs at the most recent hackathon.

Firefox takes a very different approach. It was one of the first to adopt a formal VRP program in 2004, copying Netscape's example in 1995, and initially paid $500 per serious flaw before increasing the fee to $3,000 in 2010 after Google came on the scene. It's a set fee for serious flaws.

Of the two systems, the researchers found Google's approach works best, thanks in part to the Lottery Effect. Just over 84 per cent of Google's bounty payments are $1,000 or less, but the big money events lure researchers into the field with a few very big cash prizes, and so the Chocolate Factory gets a lot more flaws reported than Firefox.

But Google gets good value for its big prizes as well, the study found. One Pwnium contest winner uncovered a flaw so serious that Google conducted a full review of the Chrome kernel file API and found a rat's nest of other vulnerabilities stemming from the same issue.

Based on the study data, being an independent bug-rustler isn't a career that's going to pay the rent on its own. One Firefox researcher earned $141,000 over the three-year period and three Chrome flaw-finders made $80,000 apiece, but only six and five people respectively made $20,000 or more.

"Contributing to a single VRP is, in general, not a viable full-time job, though contributing to multiple VRPs may be, especially for unusually successful vulnerability researchers," the team hypothesizes.

However, it may be a route to full-time employment for those looking to get into the field. Successful flaw finders get noticed and hired by firms that specialize in the field, or by the browser manufacturers themselves, the team suggests.

Meanwhile El Reg suggests that Apple and Microsoft take a good long look at the study. Microsoft has recently broken its bug-bounty virginity with a $150,000 hacking contest at this year's Black Hat security conference, but there's no sign that Apple's likely to budge on its longstanding policy of not paying for bugs. ®

Designing a Defense for Mobile Applications

More from The Register

next story
Secure microkernel that uses maths to be 'bug free' goes open source
Hacker-repelling, drone-protecting code will soon be yours to tweak as you see fit
How long is too long to wait for a security fix?
Synology finally patches OpenSSL bugs in Trevor's NAS
Roll out the welcome mat to hackers and crackers
Security chap pens guide to bug bounty programs that won't fail like Yahoo!'s
HIDDEN packet sniffer spy tech in MILLIONS of iPhones, iPads – expert
Don't panic though – Apple's backdoor is not wide open to all, guru tells us
Researcher sat on critical IE bugs for THREE YEARS
VUPEN waited for Pwn2Own cash while IE's sandbox leaked
Four fake Google haxbots hit YOUR WEBSITE every day
Goog the perfect ruse to slip into SEO orfice
Putin: Crack Tor for me and I'll make you a MILLIONAIRE
Russian Interior Ministry offers big pile o' roubles for busting pro-privacy browser
prev story

Whitepapers

Designing a Defense for Mobile Applications
Learn about the various considerations for defending mobile applications - from the application architecture itself to the myriad testing technologies.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Top 8 considerations to enable and simplify mobility
In this whitepaper learn how to successfully add mobile capabilities simply and cost effectively.
Seven Steps to Software Security
Seven practical steps you can begin to take today to secure your applications and prevent the damages a successful cyber-attack can cause.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.