Feeds

Data scientists: Do they even exist?

Data data everywhere, but not a drop to shrink

Internet Security Threat Report 2014

Open ... and Shut Big Data is all the rage. Now if only someone had to clue what to do with it.

According to a new survey of senior executives by Big data consultantancy NewVantage, Big Data is "top of mind for leading industry executives," but these same executives struggle to find the right people to analyse their data. In fact, while 70 per cent of those organisations surveyed plan to hire data scientists, 100 per cent of them said they find it at least "somewhat challenging" to hire competent data scientists:

Given the difficulty in finding qualified people to analyse data, it's perhaps not surprising that only 0.5 per cent of enterprise data gets analysed, according to IDC. But if this is the case, why is Big Data so big?

After all, Gartner expects Big Data to drive $34bn in IT spending in 2013. Some companies, like Sears, clearly "get" Big Data and are putting it to work. But for the unwashed masses of enterprise IT, it sounds like Big Data is an aspiration, not a reality.

Still, it's an aspiration that has hard dollars chasing it. Of the top-10 job skills in demand on Indeed.com's job boards, two of them are Big Data-related. Over time, however, I suspect this data scientist arms race to be absorbed by two other trends:

1. Big Data technology being embedded into applications and

2. Enterprises training existing employees on Big Data technologies rather than hiring data scientists.

On the first trend, Cloudera chief executive Mike Olson perhaps said it best when he argued that the value of big-data technology like Hadoop will increasingly be delivered through applications. Enterprises won't need data scientists as their applications will process and analyse the data for them. Yes, someone will still need to know which questions to ask of the data, but the hard-core science of it should be rendered simpler by applications.

The second trend is equally important, and was called out by Gartner analyst Svetlana Sicular, who posits: "Organisations already have people who know their own data better than mystical data scientists" and that: "Learning Hadoop is easier than learning the company’s business." So the focus of enterprises should be training employees to use tools like Hadoop, not to waste cycles and recruiting fees scouring the planet for mythical data scientists.

All of which should provide some comfort to those organisations that have been struggling to find data scientists to analyse their data. It may turn out that the "mythical data scientist" is actually Lily who works one cubicle over. ®

Matt Asay is vice president of corporate strategy at 10gen, the MongoDB company. Previously he was SVP of business development at Nodeable, which was acquired in October 2012. He was formerly SVP of biz dev at HTML5 start-up Strobe (now part of Facebook) and chief operating officer of Ubuntu commercial operation Canonical. With more than a decade spent in open source, Asay served as Alfresco's general manager for the Americas and vice president of business development, and he helped put Novell on its open source track. Asay is an emeritus board member of the Open Source Initiative (OSI). His column, Open...and Shut, appears three times a week on The Register. You can follow him on Twitter @mjasay.

Choosing a cloud hosting partner with confidence

More from The Register

next story
Facebook pays INFINITELY MORE UK corp tax than in 2012
Thanks for the £3k, Zuck. Doh! you're IN CREDIT. Guess not
DOUBLE BONK: Testy fanbois catch Apple Pay picking pockets
Users wail as tapcash transactions are duplicated
Happiness economics is bollocks. Oh, UK.gov just adopted it? Er ...
Opportunity doesn't knock; it costs us instead
YARR! Pirates walk the plank: DMCA magnets sink in Google results
Spaffing copyrighted stuff over the web? No search ranking for you
In the next four weeks, 100 people will decide the future of the web
While America tucks into Thanksgiving turkey, the world will be taking over the net
Microsoft EU warns: If you have ties to the US, Feds can get your data
European corps can't afford to get complacent while American Big Biz battles Uncle Sam
prev story

Whitepapers

Choosing cloud Backup services
Demystify how you can address your data protection needs in your small- to medium-sized business and select the best online backup service to meet your needs.
Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Storage capacity and performance optimization at Mizuno USA
Mizuno USA turn to Tegile storage technology to solve both their SAN and backup issues.