Feeds

Petabyte-chomping big sky telescope sucks down baby code

Beyond the MySQL frontier

The Power of One Infographic

Robert Heinlein was right to be worried. What if there really is a planet of giant, psychic, human-hating bugs out there, getting ready to hurl planet-busting rocks in our general direction? Surely we would want to know?

Luckily, big science projects such as the Large Synoptic Survey Telescope (LSST), which (when it's fully operational in 2016) will photograph the entire night sky repeatedly for 10 years, will be able to spot such genocidal asteroids - although asteroid-spotting is just one small part of the LSST's overall mission.

Two years ago we spoke to Jeff Kantor, LSST data management project manager, who described the project as "a proposed ground-based 6.7 meter effective diameter (8.4 meter primary mirror), 10 square-degree-field telescope that will provide digital imaging of faint astronomical objects across the entire sky, night after night."

I caught up with Jeff again a couple of weeks ago, and asked him how this highly ambitious project is progressing. "Very nicely" seems to be the crux of his answer.

It might not make for the most dramatic of headlines but given the scale and complexity of what's being developed, this in itself is a laudable achievement. In Jeff's words: "First, we have to process 6.4GB images every 15 seconds. As context, it would take 1,500 1080p HD monitors to display one image at full resolution.

"The images must go through a many-step pipeline in under a minute to detect transient phenomena, and then we have to notify the scientific community across the entire world about those phenomena. That will take a near real-time 3,000-core processing cluster, advanced parallel processing software, very sophisticated image processing and astronomical applications software, and gigabit/second networks.

"Next, we have to re-process all the images taken since the start of the survey every year for 10 years to generate astronomical catalogs, and before releasing them we need to quality assure the results."

That's about 5PB of image data/year, over 10 years, resulting in 50PB of image data and over 10PB of catalogs. The automated QA alone will require a 15,000-core cluster (for starters), parallel processing and database software, data mining and statistical analysis, and advanced astronomical software.

They now have a prototype system of about 200,000 lines of C++ and Python representing most of the capability needed to run an astronomical survey of the magnitude typically done today. Next, they have to scale this up to support LSST volumes. According to Jeff: "We hope to have all of that functioning at about 20 per cent of LSST scale of the end of our R&D phase. We then have six years of construction and commissioning to 'bullet-proof' and improve it, and to test it out with the real telescope and camera."

The incremental development and R&D mode the team is following could be called agile, although this is agile on a grand scale. Each year or six months, they do a new design and a new software release, called a Data Challenge. Each DC is a complete project with a plan, requirements, design, code, integration and test, and production runs.

Lessons learned

The fifth release just went out the door, and they've completely re-done their UML-based design third times with the lessons learned from each DC. They're using Enterprise Architect to develop each model, following a version of the agile ICONIX object modeling process tailored for algorithmic (rather than use case driven) development. I've co-authored a book on the ICONIX process, Use Case Driven Object Modeling with UMLTheory and Practice, here.

ICONIX uses a core subset of the UML rather than every diagram under the sun, and this leanness has allowed them to roll the content into a new model as a starting point for the next DC.

Jeff explains: "After each DC, we also extract the design/lessons learned from the DC model to the LSST Reference Design Model which is the design for the actual operational system. That last model is also used to trace up to a SysML-based model containing the LSST system-level requirements."

Eight steps to building an HP BladeSystem

More from The Register

next story
Malaysian Airlines flight MH17 claimed lives of HIV/AIDS cure scientists
Researchers, advocates, health workers among those on shot-down plane
Forty-five years ago: FOOTPRINTS FOUND ON MOON
NASA won't be back any time soon, sadly
Mwa-ha-ha-ha! Eccentric billionaire Musk gets his PRIVATE SPACEPORT
In the Lone Star State, perhaps appropriately enough
MARS NEEDS OCEANS to support life - and so do exoplanets
Just being in the Goldilocks zone doesn't mean there'll be anyone to eat the porridge
The Sun took a day off last week and made NO sunspots
Someone needs to get that lazy star cooking again before things get cold around here
Diary note: Pluto's close-up is a year from … now!
New Horizons is less than a year from the dwarf planet
Boffins discuss AI space program at hush-hush IARPA confab
IBM, MIT, plenty of others invited to fill Uncle Sam's spy toolchest, but where's Google?
prev story

Whitepapers

Reducing security risks from open source software
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Application security programs and practises
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.