Feeds

Petabyte-chomping big sky telescope sucks down baby code

Beyond the MySQL frontier

High performance access to file storage

Boffins share their data

Perhaps the most exciting aspect of LSST is the sharing of data and discoveries, allowing schools, universities and pretty much anyone to run collaborative projects, or simply to explore the latest data. I could see open-source visualization projects springing up, Typepad widgets layering sky maps over time, and so on.

But this kind of openness, for such huge amounts of data, presents its own technical challenge.

Jeff told us: "We have to allow scientists to quickly access any or all of that data, and support their science by enabling them to run their own scientific software along with our software. Some of the science hasn't even been conceived yet and, for what does exist, the algorithms will improve over the decade-long survey. So, we have to allow for both algorithmic and infrastructure evolution over 10 years."

Broadly speaking, the data will be published to the outside world at two levels: "For scientists, the data will be served from dedicated Data Access Centers in the US and Chile. There, they will have supercomputing resources available, advanced tools and user interfaces, access to grid computing resources and so on.

"At the same time, we will feed the data to our Education and Public Outreach Center, so that citizen-science programs, educational institutions, and museums can work with the data to create courses, exhibits, portals, and web applications to engage those communities."

SciDB in the frame

When it comes to storage and query in the science community, there's been some talk of boffins going "post-relational" with mega-database SciDB from relational legend Michael Stonebraker. Surely that would be suited to this kind of environment? The LSST team is evaluating SciDB - and it certainly seems like a good fit, designed as it is for large volumes of information crunched on thousands of nodes in distributed data centers. So does this mean that the LSST will be abandoning its existing MySQL server - throwing out relational data structures in favor of a less restrictive, multi-dimensional mathematical array model?

Jeff explains: "The baseline is still a custom parallelization layer (called qserv), on top of MySQL and xrootd. We have implemented a prototype of this layer and have tested it successfully on a 40-node cluster. We are currently testing it on a 100-node cluster to make sure it scales seamlessly with additional servers.

"We are evaluating SciDB, MonetDB, Hadoop, and other technologies as alternatives or complements to the baseline. We developed dbms-agnostic schema, test data, and a set of 65 standard queries that exercise and stress the database. We hope to implement these in all those technologies and run tests for side-by-side comparisons."

There have been some interesting astronomical discoveries and advances made recently - for example, discovering a new spiral arm in our Galaxy, and also discovering that the Milky Way is cube-shaped. I asked Jeff if there is anything out there - or any aspect of the Universe - that he secretly hopes the LSST will discover, or prove to be true (and naturally I hoped he would say something about detecting cosmic missiles being tossed at us by misanthropic alien bugs).

"It would be wonderful to know more about how the universe evolved and where it is headed (Cosmology). That is a primary mission for LSST and I have every reason to believe it will uncover astounding new science." ®

Matt Stephens is the founder of independent book publisher Fingerpress, and co-authored Design Driven Testing: Test Smarter, Not Harder (Apress, 2010)

High performance access to file storage

More from The Register

next story
Fancy joining Reg hack on quid-a-day challenge?
Recruiting now for charity starvation diet
Red-faced LOHAN team 'fesses up in blown SPEARS fuse fiasco
Standing in the corner, big pointy 'D' hats
KILLER SPONGES menacing California coastline
Surfers are safe, crustaceans less so
Opportunity selfie: Martian winds have given the spunky ol' rover a spring cleaning
Power levels up 70 per cent as the rover keeps on truckin'
Discovery time for 200m WONDER MATERIALS shaved from 4 MILLENNIA... to 4 years
Alloy, Alloy: Boffins in speed-classification breakthrough
Elon Musk's LEAKY THRUSTER gas stalls Space Station supply run
Helium seeps from Falcon 9 first stage, delays new legs for NASA robonaut
Top Secret US payload launched into space successfully
Clandestine NRO spacecraft sets off on its unknown mission
New FEMTO-MOON sighted BIRTHING from Saturn's RING
Icy 'Peggy' looks to be leaving the outer rings
prev story

Whitepapers

Top three mobile application threats
Learn about three of the top mobile application security threats facing businesses today and recommendations on how to mitigate the risk.
Combat fraud and increase customer satisfaction
Based on their experience using HP ArcSight Enterprise Security Manager for IT security operations, Finansbank moved to HP ArcSight ESM for fraud management.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Five 3D headsets to be won!
We were so impressed by the Durovis Dive headset we’ve asked the company to give some away to Reg readers.
SANS - Survey on application security programs
In this whitepaper learn about the state of application security programs and practices of 488 surveyed respondents, and discover how mature and effective these programs are.