Spy romp Zero Dark Thirty: The tech behind the special effects
Captain, we need those digital files, NOW!
Magic Quadrant for Enterprise Backup/Recovery
High capacity and fast data access is vital in the world of digital film processing, and its IT bods are always on the look out for their next speed and capacity fix.
Networked storage biz Avere worked with post-production house Image Engine to feed its need for speed in rendering spy thriller Zero Dark Thirty - directed by The Hurt Locker's Kathryn Bigelow.
As we've come to expect with visual effects, the files are huge, creative manpower is expensive and waiting for scenes to render is therefore costly. The faster the artists can get their frames from the storage vault and write completed work back the better.
Image Engine previously worked on special effects for movies such as District 9, The Incredible Hulk and The Thing. It has grown steadily since it was founded in 1995 and is now offering 4K resolution, with a Christie 4K DLP Cinema projector in its screening room.
The Vancouver-based house used NetApp gear for filer storage but in 2010 it deployed BlueArc hardware-accelerated networked arrays for storing digital movie files. This, in the shape of two Titan 3200 boxes, was the tech behind its work on sci-fi romp District 9, which commenced in 2008. Active projects were kept on SAS drives while applications and older projects were stored on the SATA drives. Files zipped around a 10Gbps Ethernet LAN linking workstations and the storage arrays.
Two years on and the number and size of the files involved in special effects work has increased and the storage that was sufficient for District 9 wasn't good enough for Zero Dark Thirty and other workloads.
Image Engine placed a cluster of Avere FXT 3500 Edge filers - with tiered RAM, flash and disk file storage acceleration - in front of its storage arrays to enable its artists to access their work faster. It found it was able to halve its storage costs by doing this, avoiding buying masses more disk spindles to achieve the performance it needed from its arrays. Regularly accessed data is automatically stored in the FXT device and streamed quickly to requesting workstations; the FXT acts as a super-charger for the array.
Image Engine's head of technology Gino Del Rosario said: "By leveraging Avere, we are scaling and being able to send and route massively large files without any challenges."
Rosario said the previous storage setup did not separate the scaling of performance from that of capacity, meaning if he wanted to ramp up performance he had to scale capacity as well.
He's also a fan of the availability of a simpler two-tier storage architecture: "Our cluster allows us to separate storage performance from capacity. It allows us to rethink our storage infrastructure. The ability to abstract performance gives us the ability to go with a single tier of high density/cost effective disk."
Avere CEO Ron Bianchini is keen to plug this message: "Digital effects companies provide a perfect example of the limitations of the traditional network-attached software architecture where performance and capacity can't be scaled independent of one another." But he would say that, wouldn't he.
Image Engine used analytics provided by the FXT's user interface to probe system performance and identify hotspots, IOPS and latency issues.
Digital film post-production work is a continuously moving target. When the movie house bosses discovered networked filers, they were ecstatic. Then the files grew in size and complexity and the artists cried in frustration. Then the houses found hardware-accelerated filers - such as BlueArc's - and also scale-out filers, like Isilon, and became ecstatic again.
Then the film data grew in size and complexity again, and everyone howled with frustration again. Now they've plugged in flash-based front-ends and are happy again. It won't last.
These businesses are technology junkies and will always need the latest fix at the intersection of enormous file capacity and data access performance; unless their workload complexity and file size plateaus. There's no sign of that right now and Avere is sitting pretty. But beware: all-flash arrays and cloud computing are sure to somehow invade the post-production digital realm in the future. ®
COMMENTS
Re: How big is BIG?
Final output will generally be in DPX files, which for 4k are 36MB per frame. Frame rates run at anything from 24-50FPS.
For an HFR movie like the Hobbit, that is 36MB x 48fps x 5400 seconds = just shy of 9TB for final delivery *only*.
The finalled frames will comprise any number of layers - one of the recent movies we worked on had anything up to 12 layers, all original source dpx or exrs. That's not counting any other steps like simulation cache data, point cloud data, models and textures.
As a final note - we worked on a film due to be released soon. We worked on only parts of 5 sequences in the movie. We worked for around 6 months, generating anything from 100GB-1TB of data per day, some of which is transient data. We ended up with a tape archive of the project rolling in at just short of 70TB, for around 7 minutes of total on-screen time.
In Visual FX, big data gets BIG very quickly.
Re: What about the plot?
If you're writing a script that requires a massive storage array, you may need to go back in and trim it a little.
Re: SFX
I think the CG dinosaurs look okay in the dark / rain (some scenes have CG and conventional models together such as the giant T-Rex head when they're trapped in the car which withdraws and CG takes over). It's the dinos out in daylight which look awful and fake.
I think some earlier CG movies could do with remastering, not to do a Lucas and change anything fundamentally but just to fix effects which look fake, rushed or obvious. Lord of the Rings has some pretty ropey CG too in places.
IT infrastructure monitoring strategies
Requirements Checklist for Choosing a Cloud Backup and Recovery Service Provider
Data control in the cloud
Cloud based data management
Enabling efficient data center monitoring
