Feeds

Petabyte-chomping big sky telescope sucks down baby code

Beyond the MySQL frontier

Intelligent flash storage arrays

Boffins share their data

Perhaps the most exciting aspect of LSST is the sharing of data and discoveries, allowing schools, universities and pretty much anyone to run collaborative projects, or simply to explore the latest data. I could see open-source visualization projects springing up, Typepad widgets layering sky maps over time, and so on.

But this kind of openness, for such huge amounts of data, presents its own technical challenge.

Jeff told us: "We have to allow scientists to quickly access any or all of that data, and support their science by enabling them to run their own scientific software along with our software. Some of the science hasn't even been conceived yet and, for what does exist, the algorithms will improve over the decade-long survey. So, we have to allow for both algorithmic and infrastructure evolution over 10 years."

Broadly speaking, the data will be published to the outside world at two levels: "For scientists, the data will be served from dedicated Data Access Centers in the US and Chile. There, they will have supercomputing resources available, advanced tools and user interfaces, access to grid computing resources and so on.

"At the same time, we will feed the data to our Education and Public Outreach Center, so that citizen-science programs, educational institutions, and museums can work with the data to create courses, exhibits, portals, and web applications to engage those communities."

SciDB in the frame

When it comes to storage and query in the science community, there's been some talk of boffins going "post-relational" with mega-database SciDB from relational legend Michael Stonebraker. Surely that would be suited to this kind of environment? The LSST team is evaluating SciDB - and it certainly seems like a good fit, designed as it is for large volumes of information crunched on thousands of nodes in distributed data centers. So does this mean that the LSST will be abandoning its existing MySQL server - throwing out relational data structures in favor of a less restrictive, multi-dimensional mathematical array model?

Jeff explains: "The baseline is still a custom parallelization layer (called qserv), on top of MySQL and xrootd. We have implemented a prototype of this layer and have tested it successfully on a 40-node cluster. We are currently testing it on a 100-node cluster to make sure it scales seamlessly with additional servers.

"We are evaluating SciDB, MonetDB, Hadoop, and other technologies as alternatives or complements to the baseline. We developed dbms-agnostic schema, test data, and a set of 65 standard queries that exercise and stress the database. We hope to implement these in all those technologies and run tests for side-by-side comparisons."

There have been some interesting astronomical discoveries and advances made recently - for example, discovering a new spiral arm in our Galaxy, and also discovering that the Milky Way is cube-shaped. I asked Jeff if there is anything out there - or any aspect of the Universe - that he secretly hopes the LSST will discover, or prove to be true (and naturally I hoped he would say something about detecting cosmic missiles being tossed at us by misanthropic alien bugs).

"It would be wonderful to know more about how the universe evolved and where it is headed (Cosmology). That is a primary mission for LSST and I have every reason to believe it will uncover astounding new science." ®

Matt Stephens is the founder of independent book publisher Fingerpress, and co-authored Design Driven Testing: Test Smarter, Not Harder (Apress, 2010)

Intelligent flash storage arrays

More from The Register

next story
MARS NEEDS WOMEN, claims NASA pseudo 'naut: They eat less
'Some might find this idea offensive' boffin admits
Boffins who stare at goats: I do believe they’re SHRINKING
Alpine chamois being squashed by global warming
LOHAN crash lands on CNN
Overflies Die Welt en route to lively US news vid
Comet Siding Spring revealed as flying molehill
Hiding from this space pimple isn't going to do humanity's reputation any good
Experts brand LOHAN's squeaky-clean box
Phytosanitary treatment renders Vulture 2 crate fit for export
No sail: NASA spikes Sunjammer
'Solar sail' demonstrator project binned
Carry On Cosmonaut: Willful Child is a poor taste Star Trek parody
Cringeworthy, crude and crass jokes abound in Steven Erikson’s sci-fi debut
prev story

Whitepapers

Cloud and hybrid-cloud data protection for VMware
Learn how quick and easy it is to configure backups and perform restores for VMware environments.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Three 1TB solid state scorchers up for grabs
Big SSDs can be expensive but think big and think free because you could be the lucky winner of one of three 1TB Samsung SSD 840 EVO drives that we’re giving away worth over £300 apiece.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.