Crowdsourced flaw-finding cheaper than in-house bug hunters
Study shows bug bounties for browsers beats hiring humans
A study into the once-controversial practice of vulnerability rewards programs (VRPs) – paying researchers bug bounties for reporting security flaws – has found that for browser builders, the practice is not only more effective at spotting problems that hiring code-checkers, it's also much better value for the money.
"We ﬁnd that VRPs appear to provide an economically efﬁcient mechanism for ﬁnding vulnerabilities, with a reasonable cost/beneﬁt trade-off, "the paper from the University of California, Berkeley computer science department states. "In particular, they appear to be 2-100 times more cost effective than hiring expert security researchers to find vulnerabilities."
The paper, "An Empirical Study of Vulnerability Rewards Programs" by Matthew Finifter, Devdatta Akhawe, and David Wagner, uses three years of data from two vulnerability reward programs run by Google's Chrome and Mozilla's Firefox. Google paid out $579,605 in bounties during that period; Mozilla shelled out $570,000.
This translates to between $485 and $658 a day for the two companies, while "an average North American developer" costs around $500 per day, assuming a $100,000 salary with a 50 per cent overhead for costs such as healthcare and office space.
Over the three-year period, Google paid bounties for outsiders who spotted 371 Chrome flaws, well outperforming its best internal security researcher who got 263. With Firefox, the crowd found 148 flaws, compared to just 48 for the best internal team member, although Mozilla doesn't have anything like the hiring budget of the Chocolate Factory.
In addition, the range of flaws the bounty programs produced is much more broad because of the range of people trying different techniques to find them. While this study concentrates on browsers, the researchers suggest the model works for the rest of the software industry.
"The cost/beneﬁt trade-off may vary for other types of (i.e., non-browser) software vendors; in particular, the less costly a security incident is for a vendor, the less useful we can expect a VRP to be," the study states. "Additionally, we expect that the higher-profile the software project is (among developers and security researchers), the more effective a VRP will be."
The Lottery Effect
Google decided to start a VRP program in January 2010, paying a fee of $500 for serious security holes and a special bonus $1337 reward for critical or clever flaw discoveries. But Google also offers big bucks at its Pwnium hacking contests, with $3.14159m up for grabs at the most recent hackathon.
Firefox takes a very different approach. It was one of the first to adopt a formal VRP program in 2004, copying Netscape's example in 1995, and initially paid $500 per serious flaw before increasing the fee to $3,000 in 2010 after Google came on the scene. It's a set fee for serious flaws.
Of the two systems, the researchers found Google's approach works best, thanks in part to the Lottery Effect. Just over 84 per cent of Google's bounty payments are $1,000 or less, but the big money events lure researchers into the field with a few very big cash prizes, and so the Chocolate Factory gets a lot more flaws reported than Firefox.
But Google gets good value for its big prizes as well, the study found. One Pwnium contest winner uncovered a flaw so serious that Google conducted a full review of the Chrome kernel file API and found a rat's nest of other vulnerabilities stemming from the same issue.
Based on the study data, being an independent bug-rustler isn't a career that's going to pay the rent on its own. One Firefox researcher earned $141,000 over the three-year period and three Chrome flaw-finders made $80,000 apiece, but only six and five people respectively made $20,000 or more.
"Contributing to a single VRP is, in general, not a viable full-time job, though contributing to multiple VRPs may be, especially for unusually successful vulnerability researchers," the team hypothesizes.
However, it may be a route to full-time employment for those looking to get into the field. Successful flaw finders get noticed and hired by firms that specialize in the field, or by the browser manufacturers themselves, the team suggests.
Meanwhile El Reg suggests that Apple and Microsoft take a good long look at the study. Microsoft has recently broken its bug-bounty virginity with a $150,000 hacking contest at this year's Black Hat security conference, but there's no sign that Apple's likely to budge on its longstanding policy of not paying for bugs. ®