Feeds

There’s gold in them there data mountains

IBM on Big Data

Remote control for virtualized desktops

Interview Enterprises have a problem with data: its volume, velocity and variety is growing at an alarming rate.

Analyst IDC says that there is already more data in the world than there is space to store it, and according to IBM every two days we create as much data as all the data that was created before 2003.

According to The Economist, the US retail giant Wal-Mart, for example, handles more than “one million customer transactions every hour, feeding databases estimated at more than 2.5 petabytes – the equivalent of 167 times the number of books in America’s Library of Congress.”

If enterprises have a problem coping with their own structured data, then the huge volumes of unstructured data they receive has the potential to cause an even bigger headache.

This could be video, or audio, email correspondence, internet traffic, diagnostic or spatial data – a myriad of data types that do not arise from transactional activities or fit into a relational database in a data warehouse.

Mark Thomas, data warehousing Infosphere specialist at IBM Software Group, Australia/NZ, comments: “Organisations know they have to do something with it, but no one is quite sure what they should do or how they should handle it.”

Another fine mess

Earlier this year a MarkLogic survey found that 86 per cent of respondents said that unstructured data is important to their organisation, but only 11 per cent had clear procedures and policies in place for managing unstructured data.

In addition, 80 per cent of respondents know that the amount of unstructured data will rise in the next three years, but only 24 per cent of respondents believe their current infrastructure will be able to manage it.

There is also the odd salmon in there that makes the fishing worthwhile

According to Thomas, IBM’s customers say they often don’t know what should be analysed, and find it prohibitively expensive to integrate large volumes of unstructured data.

“An awful lot of unstructured data will prove to be of no value whatsoever – the proverbial red herring. But there is also the odd salmon in there that makes the fishing worthwhile,” he says.

It is unrealistic for organisations to expect to be able to develop fixed infrastructure around unstructured data that doesn’t conform to business requirements and whose business case value has yet to be determined, he says.

Rather what is needed is a flexible Information management foundation that allows organisations to combine their structured and unstructured data, and enable intuitive and statistical analysis to create competitive advantage from a bigger range of data attributes.

The bigger picture

IBM calls this approach Big Insights. It makes use of open-source tools such as Hadoop, as well as IBM technology such as the newly acquired data warehouse appliance Netezza, to deliver a highly robust and flexible data warehouse.

The company aims to create a Big Data platform which can handle the volume of data being created.

“It’s all about extracting value and insight from the vast amount of data available,” says Thomas.

In contrast with a traditional approach, with business users determining what questions to ask and IT structuring the data to answer these questions, IBM’s Big Insights allows for iterative and exploratory analysis on structured and unstructured data.

The idea is for IT to enable creative discovery and business users exploring what questions could be asked, according to Thomas.

“You can’t create something that is based on a business requirement because the business requirement is either unknown or changes so quickly. Twitter or Facebook weren’t really well known even a couple of years ago.

“Our customers tell us they don’t know what we want from unstructured data – and it might not even be the ability to gather and store it for long, but they want the capability to extract value and insight from it and add it to what they already have.”

Winning combination

Most companies still struggle to get the most out of their internal structured data, but there are some shining exceptions, says Thomas, such as government agencies and telcos.

He also cites instances where the marrying of structured and unstructured data can be used to deliver an additional edge

For example, it can be used to combat customer churn: a business can use text analytics to identify word patterns or sentiments from unstructured data to give additional insight to predictive churn modelling.

And real-time sentiment analysis, a currently hot area of research, of political speeches through Twitter could allow a speech to flow in a number of different ways. ®

Security for virtualized datacentres

More from The Register

next story
Facebook pays INFINITELY MORE UK corp tax than in 2012
Thanks for the £3k, Zuck. Doh! you're IN CREDIT. Guess not
Big Content outs piracy hotbeds: São Paulo, Beijing ... TORONTO?
MPAA calls Canadians a bunch of bootlegging movie thieves
Google Glassholes are UNDATEABLE – HP exec
You need an emotional connection, says touchy-feely MD... We can do that
Just don't blame Bono! Apple iTunes music sales PLUMMET
Cupertino revenue hit by cheapo downloads, says report
US court SHUTS DOWN 'scammers posing as Microsoft, Facebook support staff'
Netizens allegedly duped into paying for bogus tech advice
Feds seek potential 'second Snowden' gov doc leaker – report
Hang on, Ed wasn't here when we compiled THIS document
Verizon bankrolls tech news site, bans tech's biggest stories
No agenda here. Just don't ever mention Net neutrality or spying, ok?
prev story

Whitepapers

Why cloud backup?
Combining the latest advancements in disk-based backup with secure, integrated, cloud technologies offer organizations fast and assured recovery of their critical enterprise data.
Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
New hybrid storage solutions
Tackling data challenges through emerging hybrid storage solutions that enable optimum database performance whilst managing costs and increasingly large data stores.
Getting ahead of the compliance curve
Learn about new services that make it easy to discover and manage certificates across the enterprise and how to get ahead of the compliance curve.