Feeds

Crowdsourced flaw-finding cheaper than in-house bug hunters

Study shows bug bounties for browsers beats hiring humans

5 things you didn’t know about cloud backup

A study into the once-controversial practice of vulnerability rewards programs (VRPs) – paying researchers bug bounties for reporting security flaws – has found that for browser builders, the practice is not only more effective at spotting problems that hiring code-checkers, it's also much better value for the money.

"We find that VRPs appear to provide an economically efficient mechanism for finding vulnerabilities, with a reasonable cost/benefit trade-off, "the paper from the University of California, Berkeley computer science department states. "In particular, they appear to be 2-100 times more cost effective than hiring expert security researchers to find vulnerabilities."

The paper, "An Empirical Study of Vulnerability Rewards Programs" by Matthew Finifter, Devdatta Akhawe, and David Wagner, uses three years of data from two vulnerability reward programs run by Google's Chrome and Mozilla's Firefox. Google paid out $579,605 in bounties during that period; Mozilla shelled out $570,000.

This translates to between $485 and $658 a day for the two companies, while "an average North American developer" costs around $500 per day, assuming a $100,000 salary with a 50 per cent overhead for costs such as healthcare and office space.

Over the three-year period, Google paid bounties for outsiders who spotted 371 Chrome flaws, well outperforming its best internal security researcher who got 263. With Firefox, the crowd found 148 flaws, compared to just 48 for the best internal team member, although Mozilla doesn't have anything like the hiring budget of the Chocolate Factory.

In addition, the range of flaws the bounty programs produced is much more broad because of the range of people trying different techniques to find them. While this study concentrates on browsers, the researchers suggest the model works for the rest of the software industry.

"The cost/benefit trade-off may vary for other types of (i.e., non-browser) software vendors; in particular, the less costly a security incident is for a vendor, the less useful we can expect a VRP to be," the study states. "Additionally, we expect that the higher-profile the software project is (among developers and security researchers), the more effective a VRP will be."

The Lottery Effect

Google decided to start a VRP program in January 2010, paying a fee of $500 for serious security holes and a special bonus $1337 reward for critical or clever flaw discoveries. But Google also offers big bucks at its Pwnium hacking contests, with $3.14159m up for grabs at the most recent hackathon.

Firefox takes a very different approach. It was one of the first to adopt a formal VRP program in 2004, copying Netscape's example in 1995, and initially paid $500 per serious flaw before increasing the fee to $3,000 in 2010 after Google came on the scene. It's a set fee for serious flaws.

Of the two systems, the researchers found Google's approach works best, thanks in part to the Lottery Effect. Just over 84 per cent of Google's bounty payments are $1,000 or less, but the big money events lure researchers into the field with a few very big cash prizes, and so the Chocolate Factory gets a lot more flaws reported than Firefox.

But Google gets good value for its big prizes as well, the study found. One Pwnium contest winner uncovered a flaw so serious that Google conducted a full review of the Chrome kernel file API and found a rat's nest of other vulnerabilities stemming from the same issue.

Based on the study data, being an independent bug-rustler isn't a career that's going to pay the rent on its own. One Firefox researcher earned $141,000 over the three-year period and three Chrome flaw-finders made $80,000 apiece, but only six and five people respectively made $20,000 or more.

"Contributing to a single VRP is, in general, not a viable full-time job, though contributing to multiple VRPs may be, especially for unusually successful vulnerability researchers," the team hypothesizes.

However, it may be a route to full-time employment for those looking to get into the field. Successful flaw finders get noticed and hired by firms that specialize in the field, or by the browser manufacturers themselves, the team suggests.

Meanwhile El Reg suggests that Apple and Microsoft take a good long look at the study. Microsoft has recently broken its bug-bounty virginity with a $150,000 hacking contest at this year's Black Hat security conference, but there's no sign that Apple's likely to budge on its longstanding policy of not paying for bugs. ®

Next gen security for virtualised datacentres

More from The Register

next story
Snowden on NSA's MonsterMind TERROR: It may trigger cyberwar
Plus: Syria's internet going down? That was a US cock-up
Who needs hackers? 'Password1' opens a third of all biz doors
GPU-powered pen test yields more bad news about defences and passwords
e-Borders fiasco: Brits stung for £224m after US IT giant sues UK govt
Defeat to Raytheon branded 'catastrophic result'
Chinese hackers spied on investigators of Flight MH370 - report
Classified data on flight's disappearance pinched
Microsoft cries UNINSTALL in the wake of Blue Screens of Death™
Cache crash causes contained choloric calamity
Germany 'accidentally' snooped on John Kerry and Hillary Clinton
Dragnet surveillance picks up EVERYTHING, USA, m'kay?
prev story

Whitepapers

Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Top 10 endpoint backup mistakes
Avoid the ten endpoint backup mistakes to ensure that your critical corporate data is protected and end user productivity is improved.
Top 8 considerations to enable and simplify mobility
In this whitepaper learn how to successfully add mobile capabilities simply and cost effectively.
Rethinking backup and recovery in the modern data center
Combining intelligence, operational analytics, and automation to enable efficient, data-driven IT organizations using the HP ABR approach.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.