Bring your spade: The BIG DATA Gold Rush has begun

The forty-niners rushed to California in search of gold. Some even found it. But most stayed poor and the only people who got reliably rich were people who made and sold the picks and shovels the miners needed.

Is that the way it is with big data? Will the only people to get rich be the suppliers of big data computer software, and computer and storage hardware?

How will we know? It is damnably difficult to calculate the ROI on a big data project. You have to ask questions such as: over 12 months, did the extra profit generated by the big data systems exceed the total cost of buying, installing and operating the big data systems?

That's one for the accountants.

You can ask a simpler question: over 12 months, did the extra revenue generated by the big data systems exceed the total cost of buying, installing and operating the big data systems?

But revenue is not profit. If it costs you $500,000 to generate $600,000 extra revenue from a big data system but a $400,000 shop generated $800,000 extra revenue then the big data investment would be sub-optimal. Better to build a second shop. Obvious, innit?

This is another question for the accountants. The cost-justification for big data systems is based on generating extra profit from extra revenue (or saving cost which helps the profit number.)

Contrast this to running a set of supply-chain scheduling applications that organise real-time manufacturing decisions with a build-to-order background. These apps run as virtual machines and rely on data fetched from a networked storage array. Equipping the servers with PCIe flash means the apps run in half the time, and you optimise manufacturing runs to sales better, resulting in less scrappage of materials, fewer lost customers, and pricing better matched to market conditions.

The PCIe flash cost is easier to justify than a ponderous big data system with vague, maybe-there-will-be-profits-in-the-future hopes.

Cynics say big data is just a way to get companies to buy shiny new data warehouse and business intelligence systems. Sure, there's gold in them there hills, but will you, specifically you, find it, and will there be enough of it to justify the costs, the Big Costs, of big data systems?

If every mobile phone operator has a big data system to reduce customer churn then which of them wins? My big data system is better than yours? Or is it a case of we have to have one because all of our competitors have one? That's a get-rich-quick-and-continuing-into-the-future scheme for suppliers.

Scottish comedian Billy Connolly talked to an audience^* about the Sottish referendum: "How did you enjoy the referendum?" he said. "What a load of shite. Whatever your judgment you're going to end up with politicians."

Is that how it is with the big data question? Suppliers supplying smooth sales patter to credulous business buyers? Ask them to let you pay for their horrendously costly big data system out of future profits. Do they really, really, REALLY, believe in the value of their product? Yes? Then say you'll pay them via a 10 per cent share of any extra profit their systems generate, and watch them run screaming for the hills. ®

*AS reported in Times Online (registration and payment needed).