Unpopular opinion: All Star, by Smash Mouth, is the greatest rock song ever created.
Okay, that’s a lie. But it does make for the most compelling use of DeepFake technology we’ve ever seen. You simply have not lived until you’ve seen Gerard Butler as Leonidas and the creepy Agent Smith from The Matrix singing out the lyrics.
Behold:
The above video was made using a DeepFakes program called Wave2Lip which was created using code from the paper “A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild.” It relies on facial recognition and artificial intelligence to create the illusion that these old movie characters are lip syncing along to the classic song.
According to the researchers responsible for the original paper, Prajwal K R and Rudrabha Mukhopadhyay:
Our approach generates accurate lip-sync by learning from an “already well-trained lip-sync expert”. Unlike previous works that employ only a reconstruction loss or train a discriminator in a GAN setup, we use a pre-trained discriminator that is already quite accurate at detecting lip-sync errors. We show that fine-tuning it further on the noisy generated faces hampers the discriminator’s ability to measure lip-sync, thus also affecting the generated lip shapes.
Despite the potential for misuse, DeepFakes have managed to permeate mainstream society thanks to clever applications like this. A few of the clips show serious artifacts and it’s doubtful anyone’s going to be convinced this is ‘real,’ but it doesn’t have to be. This novel use — and the easy to follow demo and tutorials available online — show just how easy it is to make fun, entertaining footage using AI and some old movie reels.
For more information check out the GitHub for Wave2Lip here and the original paper from which it draws its code here.
Huge tip of the hat to Boing Boing and Rob Beschizza for drawing our attention to this treasure.
Published September 8, 2020 — 22:58 UTC