Original URL: https://www.theregister.com/2009/12/21/avatar_storage_effects/
The Avatar storage effect
Aspera feeds 3D flick
Posted in Storage, 21st December 2009 19:05 GMT
Comment Avatar - James Cameron's 3D-enhanced sci-fi CGI space-western movie - is set to be a stunning success, and the storage industry is jumping on the bandwagon carrying its 10 feet tall, blue-skinned aliens.
Avatar is being produced by California-based LightStorm Entertainment, which has contracted out much of the CGI (Computer-Generated Imagery) work to Weta Digital in New Zealand. Lightstorm ran an all-digital virtual filming environment with performance capture technology and a Pace/Cameron Fusion 3D Camera System, invented by Avatar director James Cameron and Vincent Pace. This produced scores of information-rich, high-resolution 3D film files from the Los Angeles studio, which had to be rendered into viewable movie components. These files were stored on a clustered Isilon IQ filer setup.
A scene from Avatar
An Isilon release says: "The Avatar production generated terabytes of data in various formats, including massive digital files used in creating Avatar's all-digital, virtual filming environment, small metadata and instructional files, still frames for review, and large media files from Avid systems." The terabytes of data were created on a weekly basis and, sometimes, a daily one.
Lightstorm could shoot hundreds of takes of any given scene and then access the latest content from the Isilon IQ cluster with its global filesystem. Isilon says that new nodes were added as needed.
Well, that's terrific, except that the photo-realistic CGI rendering work was carried out by Weta Digital in New Zealand, which isn't an Isilon customer. Weta used NetApp kit to store the incoming data, then used a huge number of workstations and bladed servers - with 30,000 cores in total - to work on it. The NetApp filers were fitted with up to five 160GB DRAM cache accelerator cards in their controllers, the PAM (Performance Acceleration Modules) caches, to speed file access by the Weta creative people and the servers.
File transfer
Another storage supplier pops up here - Aspera - whose software was used to transfer the files between Los Angeles and New Zealand. Lightstorm had previously delivered the data via FTP, or “sneaker net” hand-delivered hard drives.
Delivering the large video files via conventional FTP at intercontinental distances was impractically slow and insecure. Sneaker net was time-consuming and error prone with lengthy shipping time, damaged drives, and content that had to be dubbed from a master only to be re-ingested on the receiving end.
A Lightstorm studio partner recommended Aspera Enterprise Server (AES) be used for electronic bulk file transfer. It uses FASP v2.0 transport technology, which Aspera claims eliminates FTP, HTTP and CIFS-style bottlenecks and makes file transfer faster and more secure "even over inherently unreliable international, satellite, and wireless links.
Aspera performance chart
It says FASP uses standard UDP and achieves reliability in the application layer through an approach that retransmits only the real packet loss on the channel. The claim is that, at 10 per cent packet loss, FASP achieves 90 per cent of the targeted throughput with less than 1 per cent redundant data overhead. The transfer speed automatically adapts to use the available bandwidth and protect other traffic, while also allowing the user to pre-set and change on-the-fly individual transfer priorities, speeds, and finish times as needed."
The meaning of "FASP" is probably an amalgam of fast with "P" for predictable, "F" for files" and "ASP" for Aspera; Aspera itself isn't saying.
AES also has a file streamlining technique that enables the transfer of a large directory of small files, allegedly as efficiently as sending a single large file, without pre-tarring the files, said to save hours in some cases.
Geoff Burdick, Lightstorm's VP for production services and technology, said: "The Aspera software has... [cut] the total turnaround time for content by about one half... We consistently transfer to New Zealand at 45 Mbit/s over a 45 Mbit/s link, a 15-30X improvement over standard FTP. Our operators can track the transfer throughout the process and manage the bandwidth used as needed. Additionally, we own the software, so there are no ongoing fees or penalties as we deliver more data."
It sounds like Aspera could have a new application ahead of it; transferring data files to the cloud.
3D to Sony's rescue
The Blu-ray Disk Association has devised a 3D Blue-ray spec which delivers separate views of recorded Blu-ray content to each eye in up to 1080p resolution. Sony said PS3 game consoles will be able to play back Blu-ray 3D content in 3D.
Of course viewers will need 3D specs to view the 3D spec media on Blu-ray. Step forward Sony - again - which has a licensing deal with RealD and its stereoscopic RealD Format to develop 3D glasses for its forthcoming 3D Bravia LCD televisions and PlayStations.
The hopes here are that home TV-viewing couch potatoes and gamers will take to 3D movies and games as the next big thing and power Sony, which makes movies, TVs, games consoles, Blu-ray players and disks, and, soon 3D specs, to greatly-improved revenues and a corporate recovery. If it doesn't, then Sony boss Sir Howard Stringer could turn into three dimensional, photo-realistic toast. ®