Feeds

Big Blue prototypes software for big, big data

Coping with the astronomy info-glut

Bridging the IT gap between rising business demands and ageing tools

IBM has prototyped a software architecture for the huge data demands of astronomy projects such as the SKA (square kilometer array).

One of the many problems created by a project as large as the SKA is that wherever it’s built – we’ll know next year if the South Africa bid or the Australia / New Zealand bid wins – it’s going to generate too much data to store.

With as much as Exabyte of data as its daily dump, the SKA will demand new techniques just so astronomers can use the facility’s output (cue: Mission Impossible theme).

Working with New Zealand-based radio astronomer, Dr Melanie Johnston-Hollitt from Wellington’s Victoria University, IBM has created the Information Intensive Framework prototype.

Under the framework, data will be classified into astronomical concepts, and overlaid with a guided search facility for faster data access and fewer errors. IBM says the prototype has also suggested further improvements to achieve the SKA’s performance demands.

"Undertaking research on exa-scale datasets will force radio astronomers into a new, as yet, unexplored regime of automated processing, imaging and analysis,” Dr Johnston-Hollitt said in IBM’s announcement.

“Surveys on even SKA precursor telescopes such as ASKAP and MWA are expected to produce catalogues of tens of millions of radio sources. How we organise and classify these data, which we will have in the next three years, is a significant challenge. We will need new solutions to fully realize the vast scientific potential of these datasets and it's fantastic that organisations like IBM are prepared to take up that challenge.”

If the A/NZ team wins the SKA contract, data will have to be pre-processed close to the telescopes (most of which would be in the remote north-west of Western Australia), then sent back to the Pawsey Supercomputing Centre in Perth for storage and analysis.

Since contracts are now open for the next phase of the Pawsey Centre’s implementation, The Register wouldn’t be surprised if IBM isn’t the only company trying to position itself with relevant software architectures. ®

Mobile application security vulnerability report

More from The Register

next story
Bad back? Show some spine and stop popping paracetamol
Study finds common pain-killer doesn't reduce pain or shorten recovery
Malaysian Airlines flight MH17 claimed lives of HIV/AIDS cure scientists
Researchers, advocates, health workers among those on shot-down plane
World Solar Challenge contender claims new speed record
One charge sees Sunswift travel 500kms at over 100 km/h
Mwa-ha-ha-ha! Eccentric billionaire Musk gets his PRIVATE SPACEPORT
In the Lone Star State, perhaps appropriately enough
SMELL YOU LATER, LOSERS – Dumbo tells rats, dogs... humans
Junk in the trunk? That's what people have
All those new '5G standards'? Here's the science they rely on
Radio professor tells us how wireless will get faster in the real world
The Sun took a day off last week and made NO sunspots
Someone needs to get that lazy star cooking again before things get cold around here
prev story

Whitepapers

Designing a Defense for Mobile Applications
Learn about the various considerations for defending mobile applications - from the application architecture itself to the myriad testing technologies.
How modern custom applications can spur business growth
Learn how to create, deploy and manage custom applications without consuming or expanding the need for scarce, expensive IT resources.
Reducing security risks from open source software
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.