Original URL: https://www.theregister.com/2014/01/22/amazon_open_source_investigation/
Amazon's 'schizophrenic' open source selfishness scares off potential talent, say insiders
Moles blame Bezos for paltry code sharing
Posted in Software, 22nd January 2014 12:02 GMT
Exclusive Amazon is one of the most technically influential companies operating today – but you wouldn't know it, thanks to a dearth of published research papers and negligible code contributions to the open-source projects it relies on.
This, according to multiple insiders, is becoming a problem. The corporation is described as a "black hole" because improvements and fixes for the open-source software it uses rarely see the light of day. And, we're told, that policy of secrecy comes right from the top – and it's driving talent into the arms of its rivals.
Over the past three months, The Register has interviewed numerous Amazon insiders* about the tech giant's attitude toward open-source software, and how the cash-flush business interacts with the tech community.
The impression built up is of a company that harvests code from vast fields of open-source software while obscuring its code donations and distancing itself from the wider world of computing.
"Amazon cannot exist without open source," said one former Amazonian.
"All of the standard web technologies were in there," said another ex-employee. "We had everything from Perl to Java to C++. In a number of cases where performance [was] really important, we looked at the [Linux] kernel ... We really ran the gamut of usage."
But you wouldn't know this from Amazon's public contributions to open-source efforts: as far as El Reg can tell, the internet titan has submitted patches and other improvements to very few projects. When it does contribute, it does so typically via a third party, usually an employee's personal account that is not explicitly linked to Amazon.
These code contributions include additions to Apache Hadoop, jQuery, the Linux kernel and Ruby, we understand.
"Within Amazon it was well known that [CEO] Jeff Bezos didn't think that Amazon would gain from participating in open source except in very limited ways at the fringes of its tech," said one ex-Amazonian. "Amazon really kept its code closed."
Of course, the web giant is under no obligation to share its enhancements for open-source-licensed software if it's not distributing the code beyond its walls (or if the license doesn't require it to in any case).
This secretiveness may give Amazon a competitive edge in the short term, but – crucially – there's evidence that it could be damaging the company in the long term because few of today's most talented technologists want to work at a company that shuts them off from the wider technical community.
Many companies choose not to donate code and instead involve themselves in the community in other ways, such as attending or organizing conferences and publishing academic papers, but here Amazon is withdrawn as well.
"When I arrived [at Amazon], one of the things that was very striking was very few people went to conferences except quiet observers," explained another. "We lost a number of key technical prospects in terms of hiring people."
Where employees from Google, Microsoft, Facebook, Oracle, IBM, and others regularly speak at conferences and meet-ups around the world, Amazon's employees are instead trained to fade into the background at conferences, revealing little about themselves or their employer.
Your El Reg correspondent saw this himself when he attended Symposium on Operating Systems Principles (SOSP), the influential computer-science confab, last year. That was a week spent walking the halls of the cavernous Nemacolin Woodlands Resort in Pennsylvania, brushing shoulders with luminaries from Google, Microsoft, Cambridge University, Carnegie Mellon, and others.
Though there were a large number of Amazon.com and Amazon Web Services developers there, none of them spoke up in group sessions or made explicit reference to their company. The only Amazon-related presentation your Reg writer saw was given by Peter Vosshall, a senior engineer at Amazon.com, and this was more about the culture and management of a technical organization than a disclosure of precise techniques.
This secretiveness, "comes from Jeff," claimed another source. "It's passed down in HR training and policy. It's all very clear."
Though a select few are permitted to give public talks, when they do, they disclose far less information about their company's technology than their peers. El Reg has been to numerous obscure presentations by Amazonians where the audience is left dissatisfied by the paucity of the knowledge shared.
"Amazon behaves a lot like a classified military agency," explained another ex-Amazonian.
'There is a particular schizophrenia between retail and technology'
Multiple sources have speculated to us that Amazon's secrecy comes from Jeff Bezos' professional grounding in the financial industry, where he worked in trading systems. This field is notoriously competitive and very, very hush-hush. That may have influenced his thoughts about how open Amazon should operate, as does his role in a market where he competes with retail giants such as Walmart.
But one contact argued that a taciturn approach may not be appropriate for the advanced technology Amazon has developed for its large-scale cloud computing business division, Amazon Web Services.
"In the Amazon case, there is a particular schizophrenia between retail and technology, and the retail culture dominates," explained the source. "Retail frugality is all about secrecy because margins are so small so you can't betray anything – secrecy is a dominant factor in the Amazon culture.
"It's a huge cost to the company."
The cloak-and-dagger attitude has at times been a source of tension in the executive ranks, we're told, and Amazon's chief technology officer Werner Vogels has lobbied strenuously to Bezos for further openness from Amazon.
Vogels' efforts led to a few hard-won successes, such as getting Amazonians slightly more involved in conferences and, notably, Amazon's main contribution to the academic computer-science corpus: the publication of the Dynamo paper [PDF] in 2007. (Aside from Dynamo, Amazon has published a few other academic papers, but far fewer than the hundreds published by rivals such as Google and Microsoft.)
If not for Vogels, the Dynamo paper never would have appeared, one source said. But persuading Vogels to air some of Amazon's bespoke laundry in public was tough, they said. "It was a hard argument to make," the source said.
Since then, Amazon has not published another paper describing an internal production system, though it has certainly made great technological strides. This compares with rivals such as Google and Microsoft, which regularly lift the curtain on the titanic systems built to keep their information empires humming.
"It's extraordinarily difficult to get anything out of [Amazon]," said another former employee.
Of the nine people listed as authors on the Dynamo paper, four have since gone on to other companies ranging from their own ventures to Google and other big names.
Of the sources with whom we spoke, many indicated that Amazon's lack of participation was a key reason for why people left the company – or never joined at all. This is why Amazon's strategy of maintaining secrecy may derail the e-retailer's future if it struggles to hire the best talent.
"Amazon's approach to open source was regarded by other people as ethically questionable," said one source.
"I think if you want to go on and have a research career, it is sort of a black hole," noted a prominent industry observer.
"The sense that I always had was we were heavy consumers of open source but not really much of a flywheel for generating and pushing change back out," said another former Amazonian.
"In many cases in the big companies and all the small startups, your Github profile is your resume," explained another former Amazonian. "When I look at developers that's what I'm looking for, [but] they go to Amazon and that resume stops ... It absolutely affects the quality of their hires."
"You had no portfolio you could share with the world," said another insider on life after working at Amazon. "The argument this was necessary to attract talent and to retain talent completely fell on deaf ears."
Though some sources with whom we spoke argued that Amazon has, in a sense, an ethical duty to the technology industry to share its learnings with the wider community, this argument offers little to shareholders. However, the most persuasive argument for investors is that Amazon will ultimately lose marketshare to competitors that can attract and build technology teams that are better skilled. All the anecdotal evidence we have gathered indicates that unless something changes, Amazon risks alienating its workers from the rest of the technical community.
Though we have no doubt the company will continue to be able to hire smart, capable people, we imagine that if it continues to be so closed, it may have trouble attracting the top tech talent – and in an age in which companies depend upon arcane systems conjured up by idiosyncratic people, this may have repercussions. ®
Bootnote
* Due to Amazon's stringent policies, we will not identify any of sources lest they face repercussions from the eldritch horrors of the company's HR department.