Feeds

9TB in 20 minutes? Sign me up!

IBM hails 'breakthrough' algorithm

  • alert
  • submit to reddit

Internet Security Threat Report 2014

In an experiment IBM researchers used the fourth most powerful supercomputer in the world - a Blue Gene/P system at the Forschungszentrum Julich in Germany - to validate nine terabytes of data in less than 20 minutes, without compromising accuracy. Ordinarily, using the same system, this would take more than a day. Additionally, the process used just one percent of the energy that would typically be required.

In a press release last week, IBM hailed a "breakthrough method based on a mathematical algorithm that reduces the computational complexity, costs, and energy usage for analyzing the quality of massive amounts of data by two orders of magnitude. This new method will greatly help enterprises extract and use the data more quickly and efficiently to develop more accurate and predictive models."

OK, I’m a sucker for anything that does something a lot faster – even if I don’t quite understand how it does it.

So I have to write a least a little bit about the paper that three IBM researchers submitted documenting their technique and results. I was faced with a blizzard of phrases such as “Inverse covariance matrices,” “Matrix factorizations,” “Cubit cost,” and this helpful explanation: “First, we turned to stochastic estimation of the diagonal.”

These terms, plus many others that I also don’t understand, are in just the abstract; the body of the paper seems considerably more technical and complex.

The one phrase that I fully understood was: “We stress that the techniques presented in this work are quite general and applicable to several other important applications.” And this is an important phrase, because it means that using this technique (and others that smart guys are working on right now), we’ll be able to see orders of magnitude improvement in other analytic tasks that use vast amounts of data. It’s always good to see progress.

In case you’re interested, here are some pictures of the guys who came up with it – mostly shots of them standing around looking smarter than any of us.

Beginner's guide to SSL certificates

More from The Register

next story
Docker's app containers are coming to Windows Server, says Microsoft
MS chases app deployment speeds already enjoyed by Linux devs
'Hmm, why CAN'T I run a water pipe through that rack of media servers?'
Leaving Las Vegas for Armenia kludging and Dubai dune bashing
'Urika': Cray unveils new 1,500-core big data crunching monster
6TB of DRAM, 38TB of SSD flash and 120TB of disk storage
Facebook slurps 'paste sites' for STOLEN passwords, sprinkles on hash and salt
Zuck's ad empire DOESN'T see details in plain text. Phew!
SDI wars: WTF is software defined infrastructure?
This time we play for ALL the marbles
Windows 10: Forget Cloudobile, put Security and Privacy First
But - dammit - It would be insane to say 'don't collect, because NSA'
Oracle hires former SAP exec for cloudy push
'We know Larry said cloud was gibberish, and insane, and idiotic, but...'
Symantec backs out of Backup Exec: Plans to can appliance in Jan
Will still provide support to existing customers
prev story

Whitepapers

Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Why cloud backup?
Combining the latest advancements in disk-based backup with secure, integrated, cloud technologies offer organizations fast and assured recovery of their critical enterprise data.
Win a year’s supply of chocolate
There is no techie angle to this competition so we're not going to pretend there is, but everyone loves chocolate so who cares.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Intelligent flash storage arrays
Tegile Intelligent Storage Arrays with IntelliFlash helps IT boost storage utilization and effciency while delivering unmatched storage savings and performance.