While deepfakes have now been around for a number of years, deepfake casting and acting are relatively new. Early deepfake technologies weren’t very good, used primarily in dark corners of the internet to swap celebrities into porn videos without their consent. But as deepfakes have grown increasingly realistic, more and more artists and filmmakers have begun using them in broadcast-quality productions and TV ads. This means hiring real actors for one aspect of the performance or another. Some jobs require an actor to provide “base” footage; others need a voice.
For actors, it opens up exciting creative and professional possibilities. But it also raises a host of ethical questions. “This is so new that there’s no real process or anything like that,” Burgund says. “I mean, we were just sort of making things up and flailing about.”
“Want to become Nixon?”
The first thing Panetta and Burgund did was ask both companies what kind of actor they needed to make the deepfakes work. “It was interesting not only what were the important criteria but also what weren’t,” Burgund says.
For the visuals, Canny AI specializes in video dialogue replacement, which uses an actor’s mouth movements to manipulate someone else’s mouth in existing footage. The actor, in other words, serves as a puppeteer, never to be seen in the final product. The person’s appearance, gender, age, and ethnicity don’t really matter.
But for the audio, Respeecher, which transmutes one voice into another, said it’d be easier to work with an actor who had a similar register and accent to Nixon’s. Armed with that knowledge, Panetta and Burgund began posting on various acting forums and emailing local acting groups. Their pitch: “Want to become Nixon?”
PANETTA AND BURGUND
This is how Lewis D. Wheeler, a Boston-based white male actor, found himself holed up in a studio for days listening to and repeating snippets of Nixon’s audio. There were hundreds of snippets, each only a few seconds long, “some of which weren’t even complete words,” he says.
The snippets had been taken from various Nixon speeches, much of it from his resignation. Given the grave nature of the moon disaster speech, Respeecher needed training materials that captured the same somber tone.
Wheeler’s job was to re-record each snippet in his own voice, matching the exact rhythm and intonation. These little bits were then fed into Respeecher’s algorithm to map his voice to Nixon’s. “It was pretty exhausting and pretty painstaking,” he says, “but really interesting, too, building it brick by brick.”
PANETTA AND BURGUND
The visual part of the deepfake was much more straightforward. In the archival footage that would be manipulated, Nixon had delivered the real moon landing address squarely facing the camera. Wheeler needed only to deliver its alternate, start to finish, in the same way, for the production crew to capture his mouth movements at the right angle.
This is where, as an actor, he started to find things more familiar. Ultimately his performance would be the one part of him that would make it into the final deepfake. “That was the most challenging and most rewarding,” he says. “For that, I had to really get into the mindset of, okay, what is this speech about? How do you tell the American people that this tragedy has happened?”
“How do we feel?”
On the face of it, Zach Math, a film producer and director, was working on a similar project. He’d been hired by Mischief USA, a creative agency, to direct a pair of ads for a voting rights campaign. The ads would feature deepfaked versions of North Korean leader Kim Jong-un and Russian president Vladimir Putin. But he ended up in the middle of something very different from Panetta and Burgund’s experiment.
In consultation with a deepfake artist, John Lee, the team had chosen to go the face-swapping route with the open-source software DeepFaceLab. It meant the final ad would include the actors’ bodies, so they needed to cast believable body doubles.
The ad would also include the actors’ real voices, adding an additional casting consideration. The team wanted the deepfake leaders to speak in English, though with authentic North Korean and Russian accents. So the casting director went hunting for male actors who resembled each leader in build and facial structure, matched their ethnicity, and could do convincing voice impersonations.
MISCHIEF USA
For Putin, the casting process was relatively easy. There’s an abundance of available footage of Putin delivering various speeches, providing the algorithm with plenty of training data to deepfake his face making a range of expressions. Consequently, there was more flexibility in what the actor could look like, because the deepfake could do most of the work.
But for Kim, most of the videos available showed him wearing glasses, which obscured his face and caused the algorithm to break down. Narrowing the training footage to only the videos without glasses left far fewer training samples to learn from. The resulting deepfake still looked like Kim, but his face movements looked less natural. Face-swapped onto an actor, it muted the actor’s expressions.
To counteract that, the team began running all of the actors’ casting tapes through DeepFaceLab to see which one came out looking the most convincing. To their surprise, the winner looked least like Kim physically but had the most expressive performance.
