March 9th, 2022

Behind the Scenes: Dune at DNEG

DNEG’s VFX experts reunited with director Denis Villeneuve to help deliver the cult science fiction epic “Dune”. A mythic and emotionally charged hero’s journey, Dune tells the story of Paul Atreides, a brilliant and gifted young man born into a great destiny beyond his understanding, who must travel to the most dangerous planet in the universe to ensure the future of his family and his people.

Set against the backdrop of Arrakis, a desert planet, the Visual Effects (VFX) work that helped to bring Dune’s dystopian atmosphere to life on-screen, spanned massive and meticulously detailed FX simulations, sweeping environments, full-CG vehicles, and creatures. Led by Production VFX Supervisor and two-time Oscar winner Paul Lambert, alongside DNEG VFX Supervisors Tristan Myles and Brian Connor, the DNEG team contributed to 28 sequences and nearly 1,200 VFX shots of the film’s 1,700 total.

We spoke to Dune CG Supervisor Rhys Salcombe to find out how DNEG brought Villeneuve’s immersive vision to life with VFX. And how Clarisse, as an integral part of the pipeline for both content creation and rendering, helped DNEG make science fiction look real. Rhys worked on Dune just short of two years, joining as the shoot was starting in the desert in Jordan, through the transition to working from home during the pandemic, and up to final delivery at the end of 2020. Rhys is an expert in creating CG cities for DNEG’s many award-winning VFX projects, including Star Trek Beyond, Blade Runner 2049, Godzilla King of the Monsters and now Dune

Tell us what you were responsible for in your role as CG Supervisor on Dune?

As CG supervisor I was responsible for the 3D side of the show ranging from build, through layout, FX, lighting and environments. I was lucky to be able to take a very hands on approach to the creative direction of our work, and worked closely with our Vancouver VFX Supervisor Tristan Myles. I was also the show’s point of contact with our pipeline and tech teams, and would work with them to resolve and prioritize tooling and bugfixes, and secure enough rendering resources to meet our demand.

How did DNEG tackle the VFX work on such a huge number of shots?

DNEG delivered around 1,200 final shots on Dune, with about two thirds of that work going through the Vancouver studio and the other third handled by the team in Montreal. Within Vancouver we had two units with dedicated department supervisors to focus the work even further; broadly those units broke down into the desert and worm vs the city and battle. With outliers like the Caladan & Salusa Secundus sequences shared between them. Both units were then overseen by Tristan and myself. 

Did you feel any pressure starting to work on a project with such heritage as this one? 

Of course! With any kind of iconic property like this there’s always the weight of expectation. As a life-long fan of science fiction myself, it was very special to be able to do my part in adapting one of the most influential sci-fi books ever written, and I know the crew felt the same way. Before Dune was even in full production there was excitement about the project in the office, the expected scope of work, and the director, Denis Villeneuve’s, great catalog of past films made the show something everybody wanted to be a part of. This meant that the whole crew brought their best work to the table, above and beyond what was asked of them. Of course it also meant the quality bar expectation was even higher than we normally strive for.

How was it to work with Director Denis Villeneuve? 

Denis Villeneuve’s direction of the VFX was all about making it naturalistic. It wasn’t about showing off, often quite the opposite. We were encouraged to hide things, and simplify them compositionally, and in terms of animation; strip everything back to its most simple and let the important component of the shot or element speak for itself. It was key that the VFX always look like an extension of the shot photography and be rooted in real references. For instance, our CG deserts needed to match the look of Wadi Rum (a dramatic desert wilderness in Jordan) or the aerial plates shot in the United Arab Emirates (UAE). Some shots such as in the Spice Harvester rescue sequence had foreground action shot in the desert in Jordan, and we needed to bridge that into the look of UAE aerials with CG sculpted dunes. On this project, the CG was always there to serve the photography. 

What was the pipeline used for the VFX of Dune at DNEG?

DNEG’s VFX pipeline for Dune used Maya and Zbrush for modeling, Mari and Substance for texturing, Houdini for FX work, and all look development, and the vast majority of the lighting and rendering was done in Clarisse, and finally composited in Nuke, using a lot of deep data to handle multiple layers of swirling dust, smoke and sand. We also trialed a few new technologies, including AI-based applications to produce procedural randomized textures for naturalistic sand and rock texture results. And an application for de-lighting photography, that allowed us to use the photogrammetry captured in Jordan as the base for the CG mountains surrounding Arrakeen City. 

What resolution did you render at?

About a third of our sequences had footage which was shot in anamorphic scope which we delivered at 4k, these were mainly indoor scenes, and for most of the outdoor shots we delivered at the taller IMAX aspect. A subset of around thirty shots required what we called ‘megaframe’ versions. These were shots where the composition didn’t allow for a simple center extraction of the IMAX frame to generate the widescreen one without omitting important content, so these shots were rendered at a resolution wide and tall enough to extract both framings from them, and delivered at 7k. 

Dune is set in a desert, what were some of the biggest challenges you faced working with this environment?

One unavoidable creative challenge was CG sand, and the shot of the worm swallowing the Harvester was one of the first turned over to us to use as a test bed for the look and behavior of the sand, as well as finding ways to sell the scale.The director, and Production VFX Supervisor Paul Lambert, wanted the sand to be so fine that it essentially behaved like water when disturbed, inspired by an SFX rig on set which placed a large vibrating metal plate beneath the real sand, causing it to behave as a liquid. We took this idea and expanded it out to a gigantic scale to work with ‘Wormsign’, which is described in the film as a ‘sand wave’ that signals shallow underground movement of sandworms.

The dunes started as height fields in Houdini being deformed by an animated proxy worm, and as the worm moves through an area the height fields collapse behind it leaving a deep trench. The surrounding dunes then shallow out and lose sharpness to their peaks in a series of cascades as the dunes liquify. Vellum sims were then run on top of the height fields to get an underlying movement of points to drive the more detailed sand and once the overall motion was working, smaller simulations of cascading sand were instanced onto the vellum points. This stacked layering of FX allowed for faster iteration of gross movement before we launched into high detail simulations, and allowed us to better optimize our memory usage considering we were routinely dealing with ten terabytes of sand cache data per shot. Later in the process, the idea of occasional sand explosions were introduced to add more drama, which was achieved mostly through careful placement of dunes in the height field to choreograph where the worm would hit a dune with enough force to explode the sand forward. 

What was your approach to rendering these complex sand simulations?

Rendering the sand sims was a careful and constant balancing act of volumes and points, and the end result was usually a combination of both. We needed the smoothness of a volume solution to get the liquid look Paul wanted, but with an amount of sand points mixed in to help it clearly read as sand even from an aerial perspective. There was no set recipe for volumes and points across the show as it depended so much on camera angle and scene lighting, and so our setups would always include both, with the balance tweaked to taste afterwards. FX lead Gero Grimm led the charge on a tool called UltraRes which used a custom sand shader in Houdini pulling from a VDB points library to replicate additional points onto our sand sims at render time to create more detail, and make the sand grains look even finer. We then exported the optimized preUltraRes points as VDBPoints to Clarisse and by passing the code snippet from FX to lighting were able to use the same algorithm to recreate them exactly matching the approved FX look, which could then be integrated into the shot lighting. 

There is so much incredible VFX work in Dune. Can you tell us about some of your favorite sequences and how they were brought to life?

Dune’s Sandworm Harvester Attack 

Even though the worm only appears briefly in the film in terms of screentime, we knew it loomed large over the project and was the thing many were most excited to see. After receiving an approved design from the art department, and a concept sculpt as a starting point, one of the first things we needed to do was figure out how to make it move!

We sculpted the worm in ZBrush HD geometry over the entire creature for it to fit within RAM. Baked displacements was used to generate mesh data inside Clarisse, so that all of the occlusion curvatures could be used in texturing, and much of the texturing was done procedurally within Nuke so that the textures could be batch processed, leaving more time to refine the sculpt itself. The worm’s teeth were modeled on a whale’s baleen, with hundreds of instanced teeth pulling from six base asset variants and arranged radially inside the mouth. Since there were so many teeth, the gum intersection geometry was done procedurally in Houdini and merged back into the main asset. 

Find out more: Dune: Making of the Sandworm Harvester Attack Sequence

Dune’s Arrakeen City 

When building Arrakeen City, the shots, angles and distance to camera it would be seen at were not fully defined and there was always an expectation that new shots featuring it could appear at any time, and so we built the city with a dynamic texture resolution, and solved this challenge using some unique complex shading solutions within Clarisse. 

The art department had done several concepts which described the overall feel of the city - flat, stark, windswept and featuring bizarre monolithic structures, but these were all at a fairly macro level. So the first step for us was some concept work, led by concept artist Seungjin Woo, to determine a more granular look to the city’s buildings and materials.

Build supervisor Marc Austin quickly determined that typical UDIM texture approaches would have led to massive texture calls at render time which would’ve required huge memory usage, so we used a system of tileables instead. A typical building had between 5 and 40 UDIMs, and if we’d approached it traditionally the city would’ve required 500+ UDIMs to hold the same level of detail as the tileables. Six base materials were assigned to the city, which were light on features but varied from fine to coarse detailing. Using a tool we wrote which assigned the tileables according to probability sets - which could be adjusted on the fly - ensured multiple instances of building models would receive different texture sets in a way that was procedurally varied but not random. Texel density was used to conform the scaling of all the textures.

Since the city layout could change at any time, we needed to keep some variables live in the shaders, such as occlusion, sun bleaching and wind damage driven by proximity between assets. This way, if the layout did change, the lookdev would adapt without needing to be manually updated. We also used Clarisse’s ability to store lighting information as a texture input to express a master wind direction through the city using a light, which drove textured sand accumulation, bleaching and weathering in the shader.

Dune’s Harkonnen Invasion

The Harkonnen invasion of Arrakeen and the spaceport was one of the biggest sequences done in Vancouver, and it ran the VFX gamut in terms of scope. It had enormous explosions, rigid body sims, fire & smoke columns, city destruction, lasers, shield effects both personal and ship-sized, not to mention clashing armies and the ever present sand and blowing dust. 

For the spaceport portion of the sequence in particular, it was very important that we achieved parity between the look of our explosions in Mantra and Clarisse. We needed to do as much of our final rendering in Clarisse as possible due to the interconnectedness of all the various FX layers of the shot - fire, smoke, sand, dust, RBDs (Rigid Body Dynamics) - and the fact the explosions and fire were acting as the sequence’s primary light source. The explosions also needed to be reflected off of the ships and emit light through atmospheric volumes since the whole environment is thick with dust.

Find out more: Dune: Making of the Harkonnen Invasion Sequence

Dune’s CG Desert Mouse

The mouse only appeared in a handful of shots, but one was an extreme closeup of its ear and face. So close that you can clearly make out individual hair follicles, so we knew we had our work cut out for us building the asset.

Most importantly the mouse needed to stand on top of a dune and use its ears to capture moisture in the wind, showcasing its adaptability to Arrakis’ environment. The detail that really sold the creature in the end was damage and irregularity, both in the sculpt itself - things like scarring and knicks on the ears - and in the scatter of sand grains and small pieces of debris we added to its fur in Clarisse.

Find out more: Dune: Making of the CG Desert Mouse

Lastly, can you tell us about any particular shots or sequences that would not have been possible without Clarisse?

One particular shot which deserves to be highlighted is what was called the ‘Reign of Fire’. As Paul and Jessica are being taken from Arrakeen during the invasion, a large Harkonnen Gunship - which we affectionately called the Angry Potato - appears over the city pushing its way through a cloud of smoke. It’s bristling with guns and after lowering a flare to illuminate the city, bombards it with hundreds of missiles in an arrangement reminiscent of Angel flares launched by C-130 Hercules aircraft, or white phosphorus bombs. Each missile had its own smoke trail, was its own light source, illuminating that trail as well as all others around it and the environment, and spawned an explosion and debris cache at point of contact with the city. It was a real challenge bringing this shot together in Clarisse with so many dynamic light sources interacting with dozens of FX caches, and the ability to manage gigantic scenes was invaluable. 

Our render farm in Vancouver has three memory tiers with the highest generally reserved for that stubborn frame or two that needs a bit of extra memory to push it over the finish line. This shot had around 60 passes, and they almost all needed to be rendered exclusively on these most powerful render blades. At around 700 frames long it was an absolute beast of a shot to complete.

Learn more about the making of Dune with:

Dune: Making of the Sandworm Harvester Attack Sequence

Dune: Making of the Harkonnen Invasion Sequence

Dune: Making of the CG Desert Mouse