Feeds

Data scientists: Do they even exist?

Data data everywhere, but not a drop to shrink

Application security programs and practises

Open ... and Shut Big Data is all the rage. Now if only someone had to clue what to do with it.

According to a new survey of senior executives by Big data consultantancy NewVantage, Big Data is "top of mind for leading industry executives," but these same executives struggle to find the right people to analyse their data. In fact, while 70 per cent of those organisations surveyed plan to hire data scientists, 100 per cent of them said they find it at least "somewhat challenging" to hire competent data scientists:

Given the difficulty in finding qualified people to analyse data, it's perhaps not surprising that only 0.5 per cent of enterprise data gets analysed, according to IDC. But if this is the case, why is Big Data so big?

After all, Gartner expects Big Data to drive $34bn in IT spending in 2013. Some companies, like Sears, clearly "get" Big Data and are putting it to work. But for the unwashed masses of enterprise IT, it sounds like Big Data is an aspiration, not a reality.

Still, it's an aspiration that has hard dollars chasing it. Of the top-10 job skills in demand on Indeed.com's job boards, two of them are Big Data-related. Over time, however, I suspect this data scientist arms race to be absorbed by two other trends:

1. Big Data technology being embedded into applications and

2. Enterprises training existing employees on Big Data technologies rather than hiring data scientists.

On the first trend, Cloudera chief executive Mike Olson perhaps said it best when he argued that the value of big-data technology like Hadoop will increasingly be delivered through applications. Enterprises won't need data scientists as their applications will process and analyse the data for them. Yes, someone will still need to know which questions to ask of the data, but the hard-core science of it should be rendered simpler by applications.

The second trend is equally important, and was called out by Gartner analyst Svetlana Sicular, who posits: "Organisations already have people who know their own data better than mystical data scientists" and that: "Learning Hadoop is easier than learning the company’s business." So the focus of enterprises should be training employees to use tools like Hadoop, not to waste cycles and recruiting fees scouring the planet for mythical data scientists.

All of which should provide some comfort to those organisations that have been struggling to find data scientists to analyse their data. It may turn out that the "mythical data scientist" is actually Lily who works one cubicle over. ®

Matt Asay is vice president of corporate strategy at 10gen, the MongoDB company. Previously he was SVP of business development at Nodeable, which was acquired in October 2012. He was formerly SVP of biz dev at HTML5 start-up Strobe (now part of Facebook) and chief operating officer of Ubuntu commercial operation Canonical. With more than a decade spent in open source, Asay served as Alfresco's general manager for the Americas and vice president of business development, and he helped put Novell on its open source track. Asay is an emeritus board member of the Open Source Initiative (OSI). His column, Open...and Shut, appears three times a week on The Register. You can follow him on Twitter @mjasay.

Application security programs and practises

More from The Register

next story
BBC goes offline in MASSIVE COCKUP: Stephen Fry partly muzzled
Auntie tight-lipped as major outage rolls on
You! Pirate! Stop pirating, or we shall admonish you politely. Repeatedly, if necessary
And we shall go about telling people you smell. No, not really
Airbus promises Wi-Fi – yay – and 3D movies (meh) in new A330
If the person in front reclines their seat, this could get interesting
UK Parliament rubber-stamps EMERGENCY data grab 'n' keep bill
Just 49 MPs oppose Drip's rushed timetable
There's NOTHING on TV in Europe – American video DOMINATES
Even France's mega subsidies don't stop US content onslaught
Samsung threatens to cut ties with supplier over child labour allegations
Vows to uphold 'zero tolerance' policy on underage workers
Dude, you're getting a Dell – with BITCOIN: IT giant slurps cryptocash
1. Buy PC with Bitcoin. 2. Mine more coins. 3. Goto step 1
ITC: Seagate and LSI can infringe Realtek patents because Realtek isn't in the US
Land of the (get off scot) free, when it's a foreign owner
prev story

Whitepapers

Reducing security risks from open source software
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Application security programs and practises
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.