Feeds

Crowdsourced flaw-finding cheaper than in-house bug hunters

Study shows bug bounties for browsers beats hiring humans

SANS - Survey on application security programs

A study into the once-controversial practice of vulnerability rewards programs (VRPs) – paying researchers bug bounties for reporting security flaws – has found that for browser builders, the practice is not only more effective at spotting problems that hiring code-checkers, it's also much better value for the money.

"We find that VRPs appear to provide an economically efficient mechanism for finding vulnerabilities, with a reasonable cost/benefit trade-off, "the paper from the University of California, Berkeley computer science department states. "In particular, they appear to be 2-100 times more cost effective than hiring expert security researchers to find vulnerabilities."

The paper, "An Empirical Study of Vulnerability Rewards Programs" by Matthew Finifter, Devdatta Akhawe, and David Wagner, uses three years of data from two vulnerability reward programs run by Google's Chrome and Mozilla's Firefox. Google paid out $579,605 in bounties during that period; Mozilla shelled out $570,000.

This translates to between $485 and $658 a day for the two companies, while "an average North American developer" costs around $500 per day, assuming a $100,000 salary with a 50 per cent overhead for costs such as healthcare and office space.

Over the three-year period, Google paid bounties for outsiders who spotted 371 Chrome flaws, well outperforming its best internal security researcher who got 263. With Firefox, the crowd found 148 flaws, compared to just 48 for the best internal team member, although Mozilla doesn't have anything like the hiring budget of the Chocolate Factory.

In addition, the range of flaws the bounty programs produced is much more broad because of the range of people trying different techniques to find them. While this study concentrates on browsers, the researchers suggest the model works for the rest of the software industry.

"The cost/benefit trade-off may vary for other types of (i.e., non-browser) software vendors; in particular, the less costly a security incident is for a vendor, the less useful we can expect a VRP to be," the study states. "Additionally, we expect that the higher-profile the software project is (among developers and security researchers), the more effective a VRP will be."

The Lottery Effect

Google decided to start a VRP program in January 2010, paying a fee of $500 for serious security holes and a special bonus $1337 reward for critical or clever flaw discoveries. But Google also offers big bucks at its Pwnium hacking contests, with $3.14159m up for grabs at the most recent hackathon.

Firefox takes a very different approach. It was one of the first to adopt a formal VRP program in 2004, copying Netscape's example in 1995, and initially paid $500 per serious flaw before increasing the fee to $3,000 in 2010 after Google came on the scene. It's a set fee for serious flaws.

Of the two systems, the researchers found Google's approach works best, thanks in part to the Lottery Effect. Just over 84 per cent of Google's bounty payments are $1,000 or less, but the big money events lure researchers into the field with a few very big cash prizes, and so the Chocolate Factory gets a lot more flaws reported than Firefox.

But Google gets good value for its big prizes as well, the study found. One Pwnium contest winner uncovered a flaw so serious that Google conducted a full review of the Chrome kernel file API and found a rat's nest of other vulnerabilities stemming from the same issue.

Based on the study data, being an independent bug-rustler isn't a career that's going to pay the rent on its own. One Firefox researcher earned $141,000 over the three-year period and three Chrome flaw-finders made $80,000 apiece, but only six and five people respectively made $20,000 or more.

"Contributing to a single VRP is, in general, not a viable full-time job, though contributing to multiple VRPs may be, especially for unusually successful vulnerability researchers," the team hypothesizes.

However, it may be a route to full-time employment for those looking to get into the field. Successful flaw finders get noticed and hired by firms that specialize in the field, or by the browser manufacturers themselves, the team suggests.

Meanwhile El Reg suggests that Apple and Microsoft take a good long look at the study. Microsoft has recently broken its bug-bounty virginity with a $150,000 hacking contest at this year's Black Hat security conference, but there's no sign that Apple's likely to budge on its longstanding policy of not paying for bugs. ®

Combat fraud and increase customer satisfaction

More from The Register

next story
Parent gabfest Mumsnet hit by SSL bug: My heart bleeds, grins hacker
Natter-board tells middle-class Britain to purée its passwords
Obama allows NSA to exploit 0-days: report
If the spooks say they need it, they get it
Web data BLEEDOUT: Users to feel the pain as Heartbleed bug revealed
Vendors and ISPs have work to do updating firmware - if it's possible to fix this
Samsung Galaxy S5 fingerprint scanner hacked in just 4 DAYS
Sammy's newbie cooked slower than iPhone, also costs more to build
Mounties always get their man: Heartbleed 'hacker', 19, CUFFED
Canadian teen accused of raiding tax computers using OpenSSL bug
Snowden-inspired crypto-email service Lavaboom launches
German service pays tribute to Lavabit
One year on: diplomatic fail as Chinese APT gangs get back to work
Mandiant says past 12 months shows Beijing won't call off its hackers
prev story

Whitepapers

Designing a defence for mobile apps
In this whitepaper learn the various considerations for defending mobile applications; from the mobile application architecture itself to the myriad testing technologies needed to properly assess mobile applications risk.
3 Big data security analytics techniques
Applying these Big Data security analytics techniques can help you make your business safer by detecting attacks early, before significant damage is done.
Five 3D headsets to be won!
We were so impressed by the Durovis Dive headset we’ve asked the company to give some away to Reg readers.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Securing web applications made simple and scalable
In this whitepaper learn how automated security testing can provide a simple and scalable way to protect your web applications.