Case Study: Weta's super storage system
NetApp and StorageTek in phase one
To store hundreds of hours of digital film, Labrie went with Network Appliance F840 network-attached storage (NAS) boxes, each with 2.7 terabytes of capacity, instead of buying a giant drive array system from EMC or IBM. This is where “live” data lives, such as digital movie frames that artists need to access quickly for editing.
'Off-line' data - scans of unedited film and final production scenes - were then kept on a StorageTek L700e 40-terabyte library of Digital Linear Tape cartridges.
Labrie chose the modular NetApp boxes because they were easy to 'pop in' when huge amounts of disk space were needed suddenly. And it was; the number of artists and the disk space needed to store their work increased more than six times during the four years to 2002.
When artists are ready to turn their work into a movie scene, they send jobs to the 'rendering wall,' a cluster of 400 Linux processors.
Weta had used servers and workstations from SGI - a known supplier of high end graphics processing -
before migrating to Linux over during the 2000 to 2002 period because it is less expensive and works just as
Labrie said. “The price to put Linux in per processor is extraordinarily low, and the machines are fast as hell. With Linux you can entertain the idea of putting 400 processors in your machine room. We wouldn’t have been able to approach doing that if we had to use [SGI]; it’s almost a 10-to- 1 difference in price."
While half of the processors at Weta run Linux, moving from a rock-solid platform such as SGI’s Unix flavour (Irix) and hardware had some challenges: "One of the advantages of working with SGI is that if you have problems, you have a single person you can point at and say, ‘This is broken, fix it.’ It’s a bit more difficult with Linux." Weta had help from Red Hat and the open source community on many technical issues with Linux.
Overall, Labrie was thankful that commodity technologies - such as Gigabit Ethernet, Linux and NAS appliances - matured at the same time. "The choice to go with modularity and to stick with off-the-shelf technology was critical in our ability to deliver the film," he said.
Silicon Graphics and a virtual one-tier storage environment
Weta used a set of SGI IRIX OS-based Silicon Graphics Octane visual workstations, Silicon Graphics Onyx2 visualization systems, SGI Origin family servers, and SGI Linux OS-based visual workstations and servers. But for The Two Towers Weta upgraded the Origin 2000 server to handle twice the data load of the first film.
A key to producing the entire LOTR trilogy was an SGI product for hierarchical storage management, the SGI Data Migration Facility (DMF). It allows high-performance, reliable and efficient data management with virtually unlimited storage capability while also dramatically lowering total cost of ownership. This is achieved by moving data seamlessly between high-performance storage arrays and lower-cost-per-megabyte tape libraries.
On The Fellowship of the Ring, Weta first used DMF to manage 100TB of data from approximately 10 million files, which range from small to extremely large. A file can consist of an element, a texture, one version of a shot or a completely rendered image sequence. Adding the data from The Two Towers doubled Weta's information storage to 20 million files. Approximately 230TB, representing the first two films' worth of data, is now managed by SGI DMF.
"We rely heavily on DMF," said Scott Houston, the then Weta CTO, said in January, 2003. "DMF is running on one Origin 2000 system and we recently upgraded it to 12 400 MHz processors. The key objective is to free up as much disk space for the artists as possible."
"We use SGI DMF to offline the data from the online disk storage to tape storage."
Weta had by then decided that DLT had to be superceded by the popular LTO (Linear Tape Open) format. It had also introduced a new tier of disk storage, nearline, between the fast online drives and the offline tape storage.
Houston said: "The StorageTek L700E robotic library now has six LTO drives and four DLT drives. In June, we migrated from DLT to LTO, which gave us greater capacity on the tape cartridges and faster tape cartridges. That gave us more capacity on near-line; we went from about 25TB available to about 75TB potentially available. We're able to move data from very valuable online disk storage to near-line disk storage and still be able to retrieve it relatively quickly and seamlessly, from the artists' point of view."
One of the many challenges Weta Digital faced at the start of The Two Towers was that it needed to bring back 100,000 files from the first film from offline storage to online storage in order to make those elements, textures and shots available for the artists. Again, the totally automated DMF made that relatively painless for the artists, according to Houston.
"The 300 artists working on the second film were moving 1TB of data in and out of DMF every day. Being able to migrate the data seamlessly between online and near-line and then back again has been absolutely critical. We couldn't do that without SGI DMF," said Houston. "We need to have access to all the files, including files that are two, three, four or almost five years old, and we need to be able to have access to these as well. We keep everything, and keeping and managing that is going to be a challenge. There are also opportunities for repurposing some of those assets, and that will be essential for Weta in the future."
NetApp and nearline storage
For the 2005 film, King Kong, Weta progressed to using NetApp NearStore nearline storage and a NetApp IP SAN. Weta also used NetApp's Snapshot facility for data protection, mirroring and data restoration. Weta was awarded a NetApp Innovator of the Year prize in December 2004.
Weta's third CTO, Milton Ngan, said at the time: "NetApp helped us increase our operational efficiencies by providing IP SAN technologies that helped to better control costs. ... NetApp was the natural choice because we knew it could handle the highly data-intensive applications required for creating King Kong."
But all good things come to an end and soon Weta was looking elsewhere for some faster and more capacious storage.
BlueArc and super-charged NAS
It alighted on BlueArc's Titan 2000 super-charged NAS box with commodity drives but specialised go-faster BlueArc silicon in the Titan servers. BlueArc stated, on October 3rd: 'Owning one of the world's top supercomputers, Weta required storage that could keep up with the information generated by its massive server and render farm."
Ngan praised his latest vendor in traditional Weta style: ""Titan 2000 delivered consistent top-line performance during testing, and gives us the capacity we need as films increase in complexity and higher resolutions become the standard. Since integrating BlueArc into our systems, we've found that the system performed beyond our expectations. Titan is able to process and store the visual effects data from our more than 4,000 processors allowing for an accelerated workflow for our artists, removal of major bottlenecks, simplified management and reduced complexity and cost."
Weta tested Titan with hundreds of thousands of requests and found that the system was able to consistently sustain over 230,000 NFS requests per second. The ability to sustain this level of throughput even as more render nodes were added was an important part of Weta's buying decision.
It's a fabulous customer reference for BlueArc as newer superNAS vendors such as Isilon and Acopia spring into prominence. But it better keep enhancing Titan's capabilities as the one certain thing in this brief storage history of Weta is that it will just keep on raising the bar.
Expect 5,000 processors and a petabyte of storage to be reached soon.
[Read part 1 of this case study here.]