Let’s hear it for data scientists! Making our lives more and more frictionless

Find them, hold them

There are, however, big problems in big data land. Firstly, they (data scientists) are in short supply. There are many people who purport to be data scientists but they tend to not understand the big picture, nor have the attitude or love of building the answers to complex puzzles within big data sets.

Next, assuming you can find a data scientist, employing them and retaining them can be difficult. It is currently the “must have” thing for even medium-sized companies to have big data analytics. The problem occurs when the head honcho declares they need a 'big data' setup without fully understanding what big data is about, nor the most important thing: what questions they need answered.

Data scientists get hired and then the CxO starts asking for spreadsheets and PowerPoint presentations. Any decent data scientist will be hot footing it out of there quicker than you can say pivot table.

However, the biggest problem is privacy. Given that so much can be learned about you from so little there is the burning question of how far should data mining be able to go in order to try and sell you something? Within a few visits the sites will have a full and complete picture of you, like it or not.

Most people are prepared to give up some privacy for a highly personalised and frictionless experience, according to one data scientist that I spoke to – Kamil Bartocha, head of data science at Lastminute.com – but companies need to be aware of wide ranging privacy concerns and do all they can to maintain trust.

I concur with Kamil’s stance to a degree but when the system knows you better than yourselves and knows how to appeal to you with just what you want, is it perhaps an unfair fight.

Data science has become a force in technology, like it or not. It can be a force for driving more sales for retailers and banks as much as it can be a force for the selfless cause – drug modelling in Africa.

To quote US businessman John Wanamaker: “Half the money I spend on advertising is wasted; the trouble is I don’t know which half.”

OK, so Wanamaker also proposed the US buy Belgium in 1915 for $1m; we'll ignore that. But his quote on money will have resonance with many in business and government today, namely those spending money on activities without any tangible evidence of a return or benefit.

Big data and data scientists, therefore, raise the prospect that the children of Wanamaker can (at last) not just uncover which half of what they spend is wasted but also find new ways to make more.

And that’s got to be a good thing for the rest of us working in the next cubicle. ®