Etsy security rule #1: Don't be a jerk to devs
Attack thyself with 0days, preaches former hacker bod
Businesses should deploy bug bounty programs, phish their staff and launch intelligent attacks against their networks, Zane Lackey says.
The now chief security officer of SignalSciences ran through the experience of building and adapting Etsy's security team.
Lackey (@zanelackey) and his colleagues, who left the hipster bazaar to found SignalSciences, had to deal with the security fallout when Etsy began pushing to production 30 times a day under a continuous deployment and merged development and operations model.
"The fundamental shift is that vulnerabilities occur in all methodologies, but in continuously deployment there is no such thing as an out-of-band patch," Lackey said in a talk given at Duo Security offices in the US.
"How many people live through the days of out of band patches? There goes your weekend, right?
"You're just pushing production and you do that 20 or 30 times a day. When that vulnerability comes in is world-ending, it's just another day."
Continuous deployment mean continuous security and allowed small change sets to be confidently pushed out. To this end, Etsy introduced feature flags, ramp-ups which allow parts of code to be deployed to certain users, and A/B testing that allow users to determine the features they most like.
He said security teams must focus on incentives to attract business units, or be prepared to be ducked. Rule one was to not be "a jerk" to developers when stupid security flaws or bad code was found.
"Don't be a jerk," Lackey said. "Save it for over drinks ... the moment you are jerk to a developer, you lose them and all of their peers, and you might never know you did."
Trade-offs are also important. Not every bug needed to be painted as world-ending, and those which were of minimal threat should not stand in the way of production. Smaller flaws should be placed on a to-do list, which will keep developers on-side.
A vulnerabilities and its impact should be explained in language developers understand, not in deliberately technical security-speak, he added.
Etsy handed out bribes to developers who report flaws including gift cards and tee-shorts which he said worked "shockingly well".
The don't-be-a-jerk approach went further still with the recommendation that security teams take "the false-positive hit" and only approach developers with verified vulnerabilities. While this appeared to put the burden of sorting real bugs from the many more false ones, it ensured that developers would not simply hit delete on security emails.
He advised all organisations to strongly consider bug bounties, adding that a hall of fame was more important than monetary rewards.
Etsy's bug bounty even prior to official launch in September 2012, got its first bug report within 13 minutes.
"When someone has a world-ending bug, you find out via an inbound email rather than Pastebin," Lackey said.
Penetration tests, or "attack simulations" as Lackey prefers, should mimic different types of attacks, including those skilled attackers who use custom tools to maintain quiet persistent inside a network compared to those who loudly smash and grab data and leave.
He distinguishes between testing and simulation as the former being simple enumeration of vulnerabilities that prove compromise was possible, and the latter based on the need to show where detection mechanisms should be placed.
Some of your attackers' goal is to a complete smash and grab - as fast and as loud as they can to grab the database and go ... other's entire goal is just to maintain persistence and won't even act on any sort of objective for six months, or years.," he said. "Simulate both of those".
Attackers in simulations should be encouraged to use zero-days but maintain contact with developers in the event that a particular targeted box may be production.
The attacks run by Etsy ran over three or four iterations, rather than a single instance due to the fact that attackers will own boxes almost immediately. The increased iterations also provided better intelligence to defenders.
More details are in Lackey's talk.