Feeds

There’s gold in them there data mountains

IBM on Big Data

The essential guide to IT transformation

Interview Enterprises have a problem with data: its volume, velocity and variety is growing at an alarming rate.

Analyst IDC says that there is already more data in the world than there is space to store it, and according to IBM every two days we create as much data as all the data that was created before 2003.

According to The Economist, the US retail giant Wal-Mart, for example, handles more than “one million customer transactions every hour, feeding databases estimated at more than 2.5 petabytes – the equivalent of 167 times the number of books in America’s Library of Congress.”

If enterprises have a problem coping with their own structured data, then the huge volumes of unstructured data they receive has the potential to cause an even bigger headache.

This could be video, or audio, email correspondence, internet traffic, diagnostic or spatial data – a myriad of data types that do not arise from transactional activities or fit into a relational database in a data warehouse.

Mark Thomas, data warehousing Infosphere specialist at IBM Software Group, Australia/NZ, comments: “Organisations know they have to do something with it, but no one is quite sure what they should do or how they should handle it.”

Another fine mess

Earlier this year a MarkLogic survey found that 86 per cent of respondents said that unstructured data is important to their organisation, but only 11 per cent had clear procedures and policies in place for managing unstructured data.

In addition, 80 per cent of respondents know that the amount of unstructured data will rise in the next three years, but only 24 per cent of respondents believe their current infrastructure will be able to manage it.

There is also the odd salmon in there that makes the fishing worthwhile

According to Thomas, IBM’s customers say they often don’t know what should be analysed, and find it prohibitively expensive to integrate large volumes of unstructured data.

“An awful lot of unstructured data will prove to be of no value whatsoever – the proverbial red herring. But there is also the odd salmon in there that makes the fishing worthwhile,” he says.

It is unrealistic for organisations to expect to be able to develop fixed infrastructure around unstructured data that doesn’t conform to business requirements and whose business case value has yet to be determined, he says.

Rather what is needed is a flexible Information management foundation that allows organisations to combine their structured and unstructured data, and enable intuitive and statistical analysis to create competitive advantage from a bigger range of data attributes.

The bigger picture

IBM calls this approach Big Insights. It makes use of open-source tools such as Hadoop, as well as IBM technology such as the newly acquired data warehouse appliance Netezza, to deliver a highly robust and flexible data warehouse.

The company aims to create a Big Data platform which can handle the volume of data being created.

“It’s all about extracting value and insight from the vast amount of data available,” says Thomas.

In contrast with a traditional approach, with business users determining what questions to ask and IT structuring the data to answer these questions, IBM’s Big Insights allows for iterative and exploratory analysis on structured and unstructured data.

The idea is for IT to enable creative discovery and business users exploring what questions could be asked, according to Thomas.

“You can’t create something that is based on a business requirement because the business requirement is either unknown or changes so quickly. Twitter or Facebook weren’t really well known even a couple of years ago.

“Our customers tell us they don’t know what we want from unstructured data – and it might not even be the ability to gather and store it for long, but they want the capability to extract value and insight from it and add it to what they already have.”

Winning combination

Most companies still struggle to get the most out of their internal structured data, but there are some shining exceptions, says Thomas, such as government agencies and telcos.

He also cites instances where the marrying of structured and unstructured data can be used to deliver an additional edge

For example, it can be used to combat customer churn: a business can use text analytics to identify word patterns or sentiments from unstructured data to give additional insight to predictive churn modelling.

And real-time sentiment analysis, a currently hot area of research, of political speeches through Twitter could allow a speech to flow in a number of different ways. ®

Build a business case: developing custom apps

More from The Register

next story
Assange™: Hey world, I'M STILL HERE, ignore that Snowden guy
Press conference: ME ME ME ME ME ME ME (cont'd pg 94)
Premier League wants to PURGE ALL FOOTIE GIFs from social media
Not paying Murdoch? You're gonna get a right LEGALLING - thanks to automated software
Caught red-handed: UK cops, PCSOs, specials behaving badly… on social media
No Mr Fuzz, don't ask a crime victim to be your pal on Facebook
Online tat bazaar eBay coughs to YET ANOTHER outage
Web-based flea market struck dumb by size and scale of fail
Amazon takes swipe at PayPal, Square with card reader for mobes
Etailer plans to undercut rivals with low transaction fee offer
US regulators OK sale of IBM's x86 server biz to Lenovo
Now all that remains is for gov't offices to ban the boxes
XBOX One will learn to play media from USB and DLNA sources
Hang on? Aren't those file formats you hardly ever see outside torrents?
Class war! Wikipedia's workers revolt again
Bourgeois paper-shufflers have 'suspended democracy', sniff unpaid proles
prev story

Whitepapers

Endpoint data privacy in the cloud is easier than you think
Innovations in encryption and storage resolve issues of data privacy and key requirements for companies to look for in a solution.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Top 8 considerations to enable and simplify mobility
In this whitepaper learn how to successfully add mobile capabilities simply and cost effectively.
Solving today's distributed Big Data backup challenges
Enable IT efficiency and allow a firm to access and reuse corporate information for competitive advantage, ultimately changing business outcomes.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.