Feeds

Big Blue prototypes software for big, big data

Coping with the astronomy info-glut

Intelligent flash storage arrays

IBM has prototyped a software architecture for the huge data demands of astronomy projects such as the SKA (square kilometer array).

One of the many problems created by a project as large as the SKA is that wherever it’s built – we’ll know next year if the South Africa bid or the Australia / New Zealand bid wins – it’s going to generate too much data to store.

With as much as Exabyte of data as its daily dump, the SKA will demand new techniques just so astronomers can use the facility’s output (cue: Mission Impossible theme).

Working with New Zealand-based radio astronomer, Dr Melanie Johnston-Hollitt from Wellington’s Victoria University, IBM has created the Information Intensive Framework prototype.

Under the framework, data will be classified into astronomical concepts, and overlaid with a guided search facility for faster data access and fewer errors. IBM says the prototype has also suggested further improvements to achieve the SKA’s performance demands.

"Undertaking research on exa-scale datasets will force radio astronomers into a new, as yet, unexplored regime of automated processing, imaging and analysis,” Dr Johnston-Hollitt said in IBM’s announcement.

“Surveys on even SKA precursor telescopes such as ASKAP and MWA are expected to produce catalogues of tens of millions of radio sources. How we organise and classify these data, which we will have in the next three years, is a significant challenge. We will need new solutions to fully realize the vast scientific potential of these datasets and it's fantastic that organisations like IBM are prepared to take up that challenge.”

If the A/NZ team wins the SKA contract, data will have to be pre-processed close to the telescopes (most of which would be in the remote north-west of Western Australia), then sent back to the Pawsey Supercomputing Centre in Perth for storage and analysis.

Since contracts are now open for the next phase of the Pawsey Centre’s implementation, The Register wouldn’t be surprised if IBM isn’t the only company trying to position itself with relevant software architectures. ®

Providing a secure and efficient Helpdesk

More from The Register

next story
MARS NEEDS WOMEN, claims NASA pseudo 'naut: They eat less
'Some might find this idea offensive' boffin admits
SECRET U.S. 'SPACE WARPLANE' set to return from SPY MISSION
Robot minishuttle X-37B returns after almost 2 years in orbit
LOHAN crash lands on CNN
Overflies Die Welt en route to lively US news vid
You can crunch it all you like, but the answer is NOT always in the data
Hear that, 'data journalists'? Our analytics prof holds forth
Experts brand LOHAN's squeaky-clean box
Phytosanitary treatment renders Vulture 2 crate fit for export
No sail: NASA spikes Sunjammer
'Solar sail' demonstrator project binned
America's super-secret X-37B plane returns to Earth after nearly TWO YEARS aloft
674 days in space for US Air Force's mystery orbital vehicle
Carry On Cosmonaut: Willful Child is a poor taste Star Trek parody
Cringeworthy, crude and crass jokes abound in Steven Erikson’s sci-fi debut
Origins of SEXUAL INTERCOURSE fished out of SCOTTISH LAKE
Fossil find proves it first happened 385 million years ago
prev story

Whitepapers

Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Cloud and hybrid-cloud data protection for VMware
Learn how quick and easy it is to configure backups and perform restores for VMware environments.
Three 1TB solid state scorchers up for grabs
Big SSDs can be expensive but think big and think free because you could be the lucky winner of one of three 1TB Samsung SSD 840 EVO drives that we’re giving away worth over £300 apiece.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.