Related topics

Petabyte-chomping big sky telescope sucks down baby code

Beyond the MySQL frontier

Boffins share their data

Perhaps the most exciting aspect of LSST is the sharing of data and discoveries, allowing schools, universities and pretty much anyone to run collaborative projects, or simply to explore the latest data. I could see open-source visualization projects springing up, Typepad widgets layering sky maps over time, and so on.

But this kind of openness, for such huge amounts of data, presents its own technical challenge.

Jeff told us: "We have to allow scientists to quickly access any or all of that data, and support their science by enabling them to run their own scientific software along with our software. Some of the science hasn't even been conceived yet and, for what does exist, the algorithms will improve over the decade-long survey. So, we have to allow for both algorithmic and infrastructure evolution over 10 years."

Broadly speaking, the data will be published to the outside world at two levels: "For scientists, the data will be served from dedicated Data Access Centers in the US and Chile. There, they will have supercomputing resources available, advanced tools and user interfaces, access to grid computing resources and so on.

"At the same time, we will feed the data to our Education and Public Outreach Center, so that citizen-science programs, educational institutions, and museums can work with the data to create courses, exhibits, portals, and web applications to engage those communities."

SciDB in the frame

When it comes to storage and query in the science community, there's been some talk of boffins going "post-relational" with mega-database SciDB from relational legend Michael Stonebraker. Surely that would be suited to this kind of environment? The LSST team is evaluating SciDB - and it certainly seems like a good fit, designed as it is for large volumes of information crunched on thousands of nodes in distributed data centers. So does this mean that the LSST will be abandoning its existing MySQL server - throwing out relational data structures in favor of a less restrictive, multi-dimensional mathematical array model?

Jeff explains: "The baseline is still a custom parallelization layer (called qserv), on top of MySQL and xrootd. We have implemented a prototype of this layer and have tested it successfully on a 40-node cluster. We are currently testing it on a 100-node cluster to make sure it scales seamlessly with additional servers.

"We are evaluating SciDB, MonetDB, Hadoop, and other technologies as alternatives or complements to the baseline. We developed dbms-agnostic schema, test data, and a set of 65 standard queries that exercise and stress the database. We hope to implement these in all those technologies and run tests for side-by-side comparisons."

There have been some interesting astronomical discoveries and advances made recently - for example, discovering a new spiral arm in our Galaxy, and also discovering that the Milky Way is cube-shaped. I asked Jeff if there is anything out there - or any aspect of the Universe - that he secretly hopes the LSST will discover, or prove to be true (and naturally I hoped he would say something about detecting cosmic missiles being tossed at us by misanthropic alien bugs).

"It would be wonderful to know more about how the universe evolved and where it is headed (Cosmology). That is a primary mission for LSST and I have every reason to believe it will uncover astounding new science." ®

Matt Stephens is the founder of independent book publisher Fingerpress, and co-authored Design Driven Testing: Test Smarter, Not Harder (Apress, 2010)

Sponsored: Network DDoS protection