Feeds

Petabyte-chomping big sky telescope sucks down baby code

Beyond the MySQL frontier

Choosing a cloud hosting partner with confidence

Boffins share their data

Perhaps the most exciting aspect of LSST is the sharing of data and discoveries, allowing schools, universities and pretty much anyone to run collaborative projects, or simply to explore the latest data. I could see open-source visualization projects springing up, Typepad widgets layering sky maps over time, and so on.

But this kind of openness, for such huge amounts of data, presents its own technical challenge.

Jeff told us: "We have to allow scientists to quickly access any or all of that data, and support their science by enabling them to run their own scientific software along with our software. Some of the science hasn't even been conceived yet and, for what does exist, the algorithms will improve over the decade-long survey. So, we have to allow for both algorithmic and infrastructure evolution over 10 years."

Broadly speaking, the data will be published to the outside world at two levels: "For scientists, the data will be served from dedicated Data Access Centers in the US and Chile. There, they will have supercomputing resources available, advanced tools and user interfaces, access to grid computing resources and so on.

"At the same time, we will feed the data to our Education and Public Outreach Center, so that citizen-science programs, educational institutions, and museums can work with the data to create courses, exhibits, portals, and web applications to engage those communities."

SciDB in the frame

When it comes to storage and query in the science community, there's been some talk of boffins going "post-relational" with mega-database SciDB from relational legend Michael Stonebraker. Surely that would be suited to this kind of environment? The LSST team is evaluating SciDB - and it certainly seems like a good fit, designed as it is for large volumes of information crunched on thousands of nodes in distributed data centers. So does this mean that the LSST will be abandoning its existing MySQL server - throwing out relational data structures in favor of a less restrictive, multi-dimensional mathematical array model?

Jeff explains: "The baseline is still a custom parallelization layer (called qserv), on top of MySQL and xrootd. We have implemented a prototype of this layer and have tested it successfully on a 40-node cluster. We are currently testing it on a 100-node cluster to make sure it scales seamlessly with additional servers.

"We are evaluating SciDB, MonetDB, Hadoop, and other technologies as alternatives or complements to the baseline. We developed dbms-agnostic schema, test data, and a set of 65 standard queries that exercise and stress the database. We hope to implement these in all those technologies and run tests for side-by-side comparisons."

There have been some interesting astronomical discoveries and advances made recently - for example, discovering a new spiral arm in our Galaxy, and also discovering that the Milky Way is cube-shaped. I asked Jeff if there is anything out there - or any aspect of the Universe - that he secretly hopes the LSST will discover, or prove to be true (and naturally I hoped he would say something about detecting cosmic missiles being tossed at us by misanthropic alien bugs).

"It would be wonderful to know more about how the universe evolved and where it is headed (Cosmology). That is a primary mission for LSST and I have every reason to believe it will uncover astounding new science." ®

Matt Stephens is the founder of independent book publisher Fingerpress, and co-authored Design Driven Testing: Test Smarter, Not Harder (Apress, 2010)

Secure remote control for conventional and virtual desktops

More from The Register

next story
GRAV WAVE DRAMA: 'Big Bang echo' may have been grit on the scanner – boffins
Exit Planet Dust on faster-than-light expansion of universe
Mine Bitcoins with PENCIL and PAPER
Forget Sudoku, crunch SHA-256 algos
SpaceX Dragon cargo truck flies 3D printer to ISS: Clawdown in 3, 2...
Craft berths at space station with supplies, experiments, toys
'This BITE MARK is a SMOKING GUN': Boffins probe ancient assault
Tooth embedded in thigh bone may tell who pulled the trigger
DOLPHINS SMELL MAGNETS – did we hear that right, boffins?
Xavier's School for Gifted Magnetotaceans
Big dinosaur wowed females with its ENORMOUS HOOTER
That's right, Doris, I've got biggest snout in the prehistoric world
Japanese volcano eruption reportedly leaves 31 people presumed dead
Hopes fade of finding survivors on Mount Ontake
That glass of water you just drank? It was OLDER than the SUN
One MEELLION years older. Some of it anyway
Canberra drone team dances a samba in Outback Challenge
CSIRO's 'missing bushwalker' found and watered
prev story

Whitepapers

Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Storage capacity and performance optimization at Mizuno USA
Mizuno USA turn to Tegile storage technology to solve both their SAN and backup issues.
The next step in data security
With recent increased privacy concerns and computers becoming more powerful, the chance of hackers being able to crack smaller-sized RSA keys increases.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.