Feeds

There’s gold in them there data mountains

IBM on Big Data

The essential guide to IT transformation

Interview Enterprises have a problem with data: its volume, velocity and variety is growing at an alarming rate.

Analyst IDC says that there is already more data in the world than there is space to store it, and according to IBM every two days we create as much data as all the data that was created before 2003.

According to The Economist, the US retail giant Wal-Mart, for example, handles more than “one million customer transactions every hour, feeding databases estimated at more than 2.5 petabytes – the equivalent of 167 times the number of books in America’s Library of Congress.”

If enterprises have a problem coping with their own structured data, then the huge volumes of unstructured data they receive has the potential to cause an even bigger headache.

This could be video, or audio, email correspondence, internet traffic, diagnostic or spatial data – a myriad of data types that do not arise from transactional activities or fit into a relational database in a data warehouse.

Mark Thomas, data warehousing Infosphere specialist at IBM Software Group, Australia/NZ, comments: “Organisations know they have to do something with it, but no one is quite sure what they should do or how they should handle it.”

Another fine mess

Earlier this year a MarkLogic survey found that 86 per cent of respondents said that unstructured data is important to their organisation, but only 11 per cent had clear procedures and policies in place for managing unstructured data.

In addition, 80 per cent of respondents know that the amount of unstructured data will rise in the next three years, but only 24 per cent of respondents believe their current infrastructure will be able to manage it.

There is also the odd salmon in there that makes the fishing worthwhile

According to Thomas, IBM’s customers say they often don’t know what should be analysed, and find it prohibitively expensive to integrate large volumes of unstructured data.

“An awful lot of unstructured data will prove to be of no value whatsoever – the proverbial red herring. But there is also the odd salmon in there that makes the fishing worthwhile,” he says.

It is unrealistic for organisations to expect to be able to develop fixed infrastructure around unstructured data that doesn’t conform to business requirements and whose business case value has yet to be determined, he says.

Rather what is needed is a flexible Information management foundation that allows organisations to combine their structured and unstructured data, and enable intuitive and statistical analysis to create competitive advantage from a bigger range of data attributes.

The bigger picture

IBM calls this approach Big Insights. It makes use of open-source tools such as Hadoop, as well as IBM technology such as the newly acquired data warehouse appliance Netezza, to deliver a highly robust and flexible data warehouse.

The company aims to create a Big Data platform which can handle the volume of data being created.

“It’s all about extracting value and insight from the vast amount of data available,” says Thomas.

In contrast with a traditional approach, with business users determining what questions to ask and IT structuring the data to answer these questions, IBM’s Big Insights allows for iterative and exploratory analysis on structured and unstructured data.

The idea is for IT to enable creative discovery and business users exploring what questions could be asked, according to Thomas.

“You can’t create something that is based on a business requirement because the business requirement is either unknown or changes so quickly. Twitter or Facebook weren’t really well known even a couple of years ago.

“Our customers tell us they don’t know what we want from unstructured data – and it might not even be the ability to gather and store it for long, but they want the capability to extract value and insight from it and add it to what they already have.”

Winning combination

Most companies still struggle to get the most out of their internal structured data, but there are some shining exceptions, says Thomas, such as government agencies and telcos.

He also cites instances where the marrying of structured and unstructured data can be used to deliver an additional edge

For example, it can be used to combat customer churn: a business can use text analytics to identify word patterns or sentiments from unstructured data to give additional insight to predictive churn modelling.

And real-time sentiment analysis, a currently hot area of research, of political speeches through Twitter could allow a speech to flow in a number of different ways. ®

Secure remote control for conventional and virtual desktops

More from The Register

next story
6 Obvious Reasons Why Facebook Will Ban This Article (Thank God)
Clampdown on clickbait ... and El Reg is OK with this
Kaspersky backpedals on 'done nothing wrong, nothing to fear' blather
Founder (and internet passport fan) now says privacy is precious
TROLL SLAYER Google grabs $1.3 MEEELLION in patent counter-suit
Chocolate Factory hits back at firm for suing customers
Facebook, Google and Instagram 'worse than drugs' says Miley Cyrus
Italian boffins agree with popette's theory that haters are the real wrecking balls
Sit tight, fanbois. Apple's '$400' wearable release slips into early 2015
Sources: time to put in plenty of clock-watching for' iWatch
Facebook to let stalkers unearth buried posts with mobe search
Prepare to HAUNT your pal's back catalogue
Ex-IBM CEO John Akers dies at 79
An era disrupted by the advent of the PC
prev story

Whitepapers

Endpoint data privacy in the cloud is easier than you think
Innovations in encryption and storage resolve issues of data privacy and key requirements for companies to look for in a solution.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Advanced data protection for your virtualized environments
Find a natural fit for optimizing protection for the often resource-constrained data protection process found in virtual environments.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Next gen security for virtualised datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.