How VFX Supervisor Ryan Laney Developed Game-Changing Technology For ‘Welcome to Chechnya,’ Shielding Identities Of LGBT Refugees

On Welcome to Chechnya, VFX supervisor Ryan Laney introduced game-changing tools with huge implications for the future of documentary filmmaking.

Directed by David France, the HBO film spotlights anti-gay purges transpiring in Chechnya in the late 2010s, following LGBT refugees as they make their way out of Russia, with the help of a network of activists.

In telling this story, France would have to address a pair of almost paradoxical drives. Because he wanted viewers to invest themselves, emotionally, in the stories of survivors, he knew he needed to give the doc a human face. At the same time, he realized early on that he would have to develop a method of disguise, which would protect the identities of his vulnerable subjects.

Ultimately, Laney would help the director to fulfill both of these objectives by placing “digital prosthetics” over the face of every survivor depicted. With queer activists in New York offering their likenesses to the production, the VFX supervisor would shoot each one, at different angles and in different lighting scenarios, against a blue screen. Then, he would translate each face into an algorithm, using deep machine learning to map it onto a particular person.

The process Laney engineered was revolutionary in a number of respects. By implementing it, filmmakers would no longer need to protect anonymity by blurring the faces of subjects, or by shooting them in total darkness.

Additionally, the artist turned the concept of “deepfake” on its head. While digitally manipulated images falling under the umbrella of “synthetic media” have quite often been used for malevolent purposes—primarily, to propagate fake news—what Laney was doing here was for the social good.

On February 9, Welcome to Chechnya made history, becoming the first documentary to make the Oscars’ visual effects shortlist. Below, its VFX supervisor expounds on the research that informed his work, the importance of “media integrity,” and his hopes for future applications of the technology he created.

DEADLINE: How did you come to connect with David France?

RYAN LANEY: David and Alice [Henty], and their team at Public Square [Films] were looking for a [visual effects] solution for about a year before they found us. They found a colleague of mine, Johnny Han, who actually worked on the project as well, and he called me up and said, “I know you like challenging problems and I think I have one for you.” So that was how we were introduced.

In the initial conversation, we didn’t actually know we were doing. The question was just, “Hey, we’ve got this thing. It’s some sensitive material, and we can’t tell you what it’s about, but it’s got to be so good that their parents won’t recognize them.” We thought that was a pretty good hypothetical, and then I guess it was about a couple of weeks later that we found out what the material was about and thought, “Oh my goodness, this has real life-and-death consequences, and we have to do a really good job at how we go about this work.”

DEADLINE: Were you involved in David’s early research and development process? I know he considered a number of different kinds of visual effects for this film.

LANEY: Yeah. They had the idea that they could rotoscope in something like a Scanner Darkly effect, so they were actually on a hunt to find somebody that could automate that. Our first reaction was, the rotoscope effect doesn’t do anything at all to hide the identity; actually, it accentuates some of the characteristics of the person.

We really started from this idea that it needed to hide identities, and were focused on that part. So, we started from this thing that wasn’t working, and all the research and development after that was us.

DEADLINE: How did you figure out an approach to creating ‘face doubles’ that would work for this film?

LANEY: The first idea that we had was actually [from] the University of Glasgow. They have a face research lab there. They did a lot of research on attractiveness and things of this nature, and it was really interesting work. But in 2013, they had published this paper, and it came with this grid of faces from around the world—and if you search for average faces of people around the world, you’ll get a hit pretty quickly.

We were really struck by this idea that, as different as people are around the world, the geometry is very similar. There’s this very subtle difference between different people, regionally and specifically, and we had this idea that we could create what you might call a “transfer function” from one person to another. Our first tests were actually just that, where we warped a picture of Obama to look more Spanish, and that was reasonably successful, to give us a hope that this idea of changing the face was going to work. But there was still this family resemblance, so we knew we had to do better.

In the first idea of doing A Scanner Darkly, we briefly looked at combining this idea of doing a subtle face transfer function, and this idea of style transfer, to create this style transfer of faces. Some other people had started in this area as well, so we knew that it was possible. We just didn’t know how we were going to do it, at film-quality resolutions, or the scale of the number of shots that we needed to do. So, most of [the software] we wrote was improving fidelity and creating style transfer for faces, and in creating a visual effects pipeline around the idea, we kind of used a traditional pipeline for most of the work. Then, this face transfer function was in the middle, where we might normally have a 3D renderer.

DEADLINE: The ‘faces’ used to shield your documentary subjects were those of activists, who volunteered their likenesses. Did you need to start with real faces to sell the effect you were going for? Or is it possible to create faces from scratch that could serve the same purpose?

LANEY: We’re a couple of years in the past at [the point we’re discussing], and now we’re on the verge of possible, if not possible. Nobody’s done it yet, but there’s some really fantastic work out of NVIDIA right now. StyleGAN2, they have a website, “This Face Does Not Exist.” The gap between that and what we need is, we need a full range of expressions for a person. Where that system is really good at generating a picture of a person, or a hypothetical person, it’s not good at creating what you might call a “temporally coherent version” of a person moving through several expressions, which is what we needed.

Our hope is eventually to be able to fabricate faces. We have a data set now of a lot of faces, and we have been able to create novel faces out of faces that we have in the data set, that are none of those people, but that are also not the person that we’re matching. So, we’re moving in the direction of being able to synthesize faces, but we still need good photography coming in, in order to know what a real face is. All of these machine-learning things really rely on good data at the outset. They’re only going to be as good as what’s coming in, so real-world photography is the best thing today.

DEADLINE: Welcome to Chechnya was shot surreptitiously on cell phones, and small cameras like GoPros. Did the kind of footage you were working with make for a bigger challenge, in mapping faces onto shots?

LANEY: Oh, absolutely. Visual effects is used to working with high-end cameras, with lots of resolution. So, we had a list of complications, and those complications involved handheld cameras, low light, high grain, poor exposure. There was a sequence that had interlaced footage. All of the things that distinguish professional camerawork from a person with a mobile phone made the footage harder to work with.

Plus, all these consumer formats record into a compressed format that has compression artifacts. So, after we put the visual effects in, we had to match the compression artifacts, so that it would sit in with the rest of the film. It was another level of that unseen level of work that makes it work. But I don’t know how [else] David and his team would have shot in any of these circumstances. I mean, they were going through checkpoints and airports, and anything that looked like a real camera would have just drawn attention and put everybody at risk.

DEADLINE: On this film, you also weren’t able to place MoCap markers on the faces of your subjects, as might be done on a typical studio project. While films like The Irishman have done away with these markers, paving a way to a future in which they’re obsolete, I imagine the absence of them here must have added another layer of complexity to your work.

LANEY: Yeah, that was all part of our decision. There was a lot in the decision-making about the treatment that we came up with, and one of those was that we had this footage that was low grade. We didn’t have tracking markers, we didn’t have good lighting, we didn’t have 3D representations of anything. So, going back to the footage, there was rolling shutter issues, which makes tracking impossible. There was a lot to what complicated it.

I feel like it’s probably important to distinguish it as different from The Irishman, which is fantastic work. We actually named the effect “FBO,” for fancy blurry oval, because we really wanted to distinguish this from a face replacement for a number of reasons. One is that there’s a really strong visual language with witness filming. Everybody knows what a silhouette shot is; everybody knows that blurry ovals, or fogging faces, indicates protection, and that the person being filmed is at risk. We wanted something that was visible, in order to create this dialogue with the audience of who in the film was most at risk, and it became part of that story, especially in a reveal at one point…

DEADLINE: When the blurry oval dissolves, and Maxim Lapunov’s real face appears…

LANEY: Exactly. So, it does sit in with the fabric of the film, without drawing attention to itself. It gives the audience this sort of human connection, because there’s so much emotion that is read in the face, through facial expressions. So, our goals were really protection, and to allow this connection between the audience and the subjects, and not actually to be a face replacement.

The other really important component of that is media integrity, and this idea of fake news and fake media. We didn’t want to be in that world, so we were very deliberate about creating something that was visible.

We actually added a halo. We got higher fidelity on the faces than we expected. They’re still not anywhere in the range of what The Irishman was, but we got better than we expected, and we ended up going back in and adding a blurry edge around everything, to be able to be clear about this dialogue with the audience, about who had the treatment, and who didn’t—from a journalistic integrity perspective, making sure that we weren’t trying to fool anybody.

Face replacements are meant to narratively fool the audience. But we wanted to be very obvious and open about [what we were doing]. We wanted to draw attention, but for the purpose of the narrative, which was that these are people at risk.

DEADLINE: Tell us more about that line between “deepfakes” and the kind of synthetic media you were creating. What distinguishes the two, from your perspective?

LANEY: We look at it as an issue of consent, and deepfakes are nonconsensual in several ways. They don’t ask the person who’s being covered, they don’t ask the person who is covering, and they’re trying to sell the audience on this fake representation of reality. In a different way, David didn’t know how, but he told the subjects in the film that he would cover them, and he actually gave them a chance to review their footage after we did the work to sign off on it. So, everybody that’s covered in the film saw the work before it was released, and felt comfortable with themselves, as they were covered. Everybody that he filmed had the option of being covered or not, and the people that we shot for the face doubles were largely volunteers from the community. Then, there’s a disclaimer at the beginning of the film, and in the first 20 shots, we played up this soft edge around everything, to acclimate the audience and show them what was going on. So, I think that’s the difference there.

DEADLINE: It must be fascinating to work in visual effects right now. Technology is evolving faster even than our ability to properly discuss it, which is why it is so important to be clear about these distinctions.

LANEY: Yeah. Tristan Harris, I don’t know if you know his name. It’s not really part of our story, but he has really influenced my thinking about, “I believe in technology. And how can I change subtly what I do in the world, so that the things I do are beneficial to humanity?”

DEADLINE: Welcome to Chechnya is quite different from any film you’ve worked on in the past. What did it mean to you to see your work used for the common good?

LANEY: It’s the most important work I’ve done to date. Getting to work on a film where lives are affected in such a way, it’ll be hard to go back to [other endeavors]. But the idea is that this is important work, and that working in the interest of human rights changes the equation. I’ve looked for jobs that are meaningful in different ways, so I think I was already in that mindset when this project came around, and I was really just fortunate to be called and asked to help out.

As far as seeing it being used in a useful way, I think every technology [can serve opposing ends]. Like, a tree branch can be a club, or a lever, or a baseball bat, or a house. We can’t imagine using tools not in a beneficial way, so we hope we’ve done a good job.

DEADLINE: How did it feel to see the film break into the Oscars’ visual effects shortlist?

LANEY: I’m floored. It’s amazing, really, to be considered at all. So, we’re really thrilled that we have been recognized, and in coming to terms with it, we realized that the Academy Awards are not about the biggest movie. They’re about moving forward the arts and sciences, and from the arts and sciences angle, we feel that we have provided a new tool to storytelling, and documentary storytelling specifically. We feel like we have done a good job at moving the needle in the arts and sciences, so we feel good about the work and are honored to be recognized by the Academy.

DEADLINE: Are there ways in which you think the software employed on this doc can be further refined? I’ve heard that you’re looking to make it available to other documentarians.

LANEY: Yeah. We actually did some rounds with MIT Documentary Lab and a company called, which educates people in shooting for witness testimony. Oh, and Co-Creation [Studio], which is another MIT group. They’re all really interested in how this is going to change the landscape, and the idea that if this will change the landscape, how do we go about doing it in an ethical way? So, all the plans that we have moving forward are maintaining this idea of good use, and we feel like the fidelity can be improved.

You mentioned the idea of making it more available: That really comes from this idea that documentaries are generally underfunded. The people that are telling the stories of the people that are most at risk in the world are also the people that are most at risk because they don’t have a voice. They don’t really have the funding.

So, we’re trying to figure out that part of the equation. But our goal really is to make this a tool that would be available to any filmmaker who needed it, and not just the big-budget films. Because people will use these same ideas for stunt double replacement; it’s already being used for that. Johnny Han has started using our capture techniques on an HBO project that he went on to after our project, so it’s already getting out there into well-funded projects. We just need to make sure that it doesn’t disappear for the underfunded projects.

Read More About:

Source: Read Full Article