Spy romp Zero Dark Thirty: The tech behind the special effects
Captain, we need those digital files, NOW!
High capacity and fast data access is vital in the world of digital film processing, and its IT bods are always on the look out for their next speed and capacity fix.
Networked storage biz Avere worked with post-production house Image Engine to feed its need for speed in rendering spy thriller Zero Dark Thirty - directed by The Hurt Locker 's Kathryn Bigelow.
As we've come to expect with visual effects, the files are huge, creative manpower is expensive and waiting for scenes to render is therefore costly. The faster the artists can get their frames from the storage vault and write completed work back the better.
Image Engine previously worked on special effects for movies such as District 9, The Incredible Hulk and The Thing. It has grown steadily since it was founded in 1995 and is now offering 4K resolution, with a Christie 4K DLP Cinema projector in its screening room.
The Vancouver-based house used NetApp gear for filer storage but in 2010 it deployed BlueArc hardware-accelerated networked arrays for storing digital movie files. This, in the shape of two Titan 3200 boxes, was the tech behind its work on sci-fi romp District 9, which commenced in 2008. Active projects were kept on SAS drives while applications and older projects were stored on the SATA drives. Files zipped around a 10Gbps Ethernet LAN linking workstations and the storage arrays.
Two years on and the number and size of the files involved in special effects work has increased and the storage that was sufficient for District 9 wasn't good enough for Zero Dark Thirty and other workloads.
Image Engine placed a cluster of Avere FXT 3500 Edge filers  - with tiered RAM, flash and disk file storage acceleration - in front of its storage arrays to enable its artists to access their work faster. It found it was able to halve its storage costs by doing this, avoiding buying masses more disk spindles to achieve the performance it needed from its arrays. Regularly accessed data is automatically stored in the FXT device and streamed quickly to requesting workstations; the FXT acts as a super-charger for the array.
Image Engine's head of technology Gino Del Rosario said: "By leveraging Avere, we are scaling and being able to send and route massively large files without any challenges."
Rosario said the previous storage setup did not separate the scaling of performance from that of capacity, meaning if he wanted to ramp up performance he had to scale capacity as well.
He's also a fan of the availability of a simpler two-tier storage architecture: "Our cluster allows us to separate storage performance from capacity. It allows us to rethink our storage infrastructure. The ability to abstract performance gives us the ability to go with a single tier of high density/cost effective disk."
Avere CEO Ron Bianchini is keen to plug this message: "Digital effects companies provide a perfect example of the limitations of the traditional network-attached software architecture where performance and capacity can't be scaled independent of one another." But he would say that, wouldn't he.
Image Engine used analytics provided by the FXT's user interface to probe system performance and identify hotspots, IOPS and latency issues.
Digital film post-production work is a continuously moving target. When the movie house bosses discovered networked filers, they were ecstatic. Then the files grew in size and complexity and the artists cried in frustration. Then the houses found hardware-accelerated filers - such as BlueArc's - and also scale-out filers, like Isilon, and became ecstatic again.
Then the film data grew in size and complexity again, and everyone howled with frustration again. Now they've plugged in flash-based front-ends and are happy again. It won't last.
These businesses are technology junkies and will always need the latest fix at the intersection of enormous file capacity and data access performance; unless their workload complexity and file size plateaus. There's no sign of that right now and Avere is sitting pretty. But beware: all-flash arrays and cloud computing are sure to somehow invade the post-production digital realm in the future. ®