Amazon parks human genome on cloud

In 1993, meat space bookseller Barnes & Noble started offering Starbucks coffee to augment customers' shopping experience. Not to be outdone, the Internet's largest bookseller has finally answered.

Amazon announced yesterday that it will allow free, easy access to hard-to-find datasets like the human genome - and a few other compilations that aren't nearly as exciting. With its new Public Data Sets initiative, Amazon hopes to commoditize access to this kind of data, while positioning its EC2 system as the preferred computing platform for researchers.

For users, this service is likely to speed up the science without digging too far into the meager stack of grant money. Amazon is currently hosting data sets from biology, chemistry, and economics, having hunted down files in the public domain. To expand the service, they are soliciting people to provide more data, provided that it's not proprietary.

They boast free access to the data. "Free" is about as relative a term as “pregnant,” but Amazon is still taking some liberty with it. While technically you don't have to pay for the data sets, the only way you can access them is by spinning up an EC2 virtual machine instance and mounting the data as an Elastic Block Store drive. Amazon charges for EC2 instances by the hour, and Elastic Block Store drives by the I/O request. Conveniently for Amazon, these data sets are equally as processable as they are large.

There are already several online directories of public domain data sets, so what's the value-add? When you run an EC2 virtual instance to access the data, you can choose a machine image that contains specialized processing tools. If parallel processing is an concern, you can spin up as many instances as you need to process the data and spin them down when you're done, only paying for the resources you use.

This is still a bit dubious, because Amazon isn't really doing any work here. The data sets and processing tools come from third parties. Bezos just uses them to sell Amazon Web Services. As far as exploiting the academic spirit of sharing and mutual betterment for profit, the Public Data Sets program is a winner.

While working with the data in Amazon's environment may be a bit faster and more convenient, the target market for this type of service is the notoriously tight-assed academic researcher. While a tenured professor may see a minor productivity increase by using Amazon Web Services, she can see an order of magnitude productivity increase by enslaving a graduate student to do the same work. Said graduate student, already living below the poverty line, is unlikely to spend money on Amazon. There is a time-honored tradition among these servile few of spending late nights in the computer lab slicing up US Census data because it's difficult to get SAS or SPSS licenses for their laptops.

Be that as it may, Amazon isn't making a big bet on this one. If the analytics tools provided on EC2 take good enough advantage of the parallelism offered by virtualization, perhaps graduate students everywhere will decide to drop a little money. Less time spent analyzing data means more time spent partying, remembering that you're still under thirty and without serious responsibility. ®

Ted Dziuba is a co-founder at Milo.com You can read his regular Reg column, Fail and You, every other Monday.