Feeds

Mapping the universe at 30 Terabytes a night

Jeff Kantor, on building and managing a 150 Petabyte database

  • alert
  • submit to reddit

Security and trust: The backbone of doing business over the internet

Kantor added: "We are also prototyping with other open-source and proprietary databases, as well as with a Map Reduce-based approach similar to that in use at Google. We are also participating in a startup venture to create a new database engine specifically oriented at large-scale databases, especially those that contain scientific and image data."

The data will be available in formats compliant with the Virtual Observatory standards, as FITS images, and as RGB images (or something equivalent).

Providing 30TB of data a day, to each and every potential user, sounds about as easy and practical as juggling elephants one-handed.

Kantor explained: "At 1Gbps, 30TB would take 67 hours to download (without overhead). That is why the Data Access Centers exist, so users can access the data and analyze it without downloading large subsets. Rather than move the data to the processing code, we permit you to process the data nearby."

One wonders how an automated system could be written to discover previously unknown classes of rare objects - part of the telescope's mission statement.

How do you program clairvoyance into a data analysis system? Kantor: "There are quite a few researchers pursuing the line that one can analyze large datasets statistically, and uncover outliers and anomalies of interest. This is very much a research topic and one that several LSST partners are pursuing.

"In addition, we are designing the software with the ability to extend it to new algorithms and data types easily. There is a tradeoff between flexibility and performance and we walk that line every day in the design."

Talking of design, agile process aficionados out there will be interested to hear that Kantor and his team are using the minimalist, UML-based ICONIX Process (a subject close to this writer's own heart) for their system and software requirements and design. The teams are geographically dispersed, so the LSST models are shared using Sparx Systems' Enterprise Architect (EA) version control integration capabilities. Individual packages are added to a central version control repository and these packages are then shared by several local EA project files.

Kantor adds: "For code, our development environment is based on the open source trac tool integrated with subversion for version control. This provides a source repository and browser, ticket system, and documentation wiki."

Measuring the success or failure of a project as massive and wide-ranging as the LSST, which will run over such a long period of time, could prove difficult. How would you know that the project is providing value for money, useful information?

Kantor agreed: "That is always a tricky question for 'big science' projects. Typically it is measured in terms of professional papers created from the survey. Additional metrics have to do with educational impact and public impact.

"The Hubble Space Telescope gave us pictures of distant objects in the Universe that changed most people's perception of and interest in astronomy. LSST has the same potential. How do you measure that?"®

Providing a secure and efficient Helpdesk

More from The Register

next story
New 'Cosmos' browser surfs the net by TXT alone
No data plan? No WiFi? No worries ... except sluggish download speed
'Windows 9' LEAK: Microsoft's playing catchup with Linux
Multiple desktops and live tiles in restored Start button star in new vids
iOS 8 release: WebGL now runs everywhere. Hurrah for 3D graphics!
HTML 5's pretty neat ... when your browser supports it
Mathematica hits the Web
Wolfram embraces the cloud, promies private cloud cut of its number-cruncher
Mozilla shutters Labs, tells nobody it's been dead for five months
Staffer's blog reveals all as projects languish on GitHub
'People have forgotten just how late the first iPhone arrived ...'
Plus: 'Google's IDEALISM is an injudicious justification for inappropriate biz practices'
SUSE Linux owner Attachmate gobbled by Micro Focus for $2.3bn
Merger will lead to mainframe and COBOL powerhouse
iOS 8 Healthkit gets a bug SO Apple KILLS it. That's real healthcare!
Not fit for purpose on day of launch, says Cupertino
prev story

Whitepapers

Secure remote control for conventional and virtual desktops
Balancing user privacy and privileged access, in accordance with compliance frameworks and legislation. Evaluating any potential remote control choice.
WIN a very cool portable ZX Spectrum
Win a one-off portable Spectrum built by legendary hardware hacker Ben Heck
Storage capacity and performance optimization at Mizuno USA
Mizuno USA turn to Tegile storage technology to solve both their SAN and backup issues.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
The next step in data security
With recent increased privacy concerns and computers becoming more powerful, the chance of hackers being able to crack smaller-sized RSA keys increases.