• Log In
  • Sign Up
    • One of the standout presentations I saw at Collision Conference here in Toronto yesterday was a presentation called "The Rise of the Digital Humans" given by Darren Hendler of Digital Domain. It was described as "Driven by new deep learning techniques realistic digital humans and characters will soon be around us in a wide variety of forms, from virtual assistants to deep fakes. The way we interact with technology and what we believe to be real will never be the same."

      Darren started out the presentation with “It’s my job to trick you that an artificial, digital human or character is really real."

      He then elaborated on the process needed to bring Thanos from AVENGERS to life.

      “When we built Thanos, we needed to understand Josh’s face… we take a whole series of scans of Josh’s face and body, certain regions like his eyes and subtle facial proportions, onto Thanos. If he puppeteers Thanos, it’s that much more believable.”

      “Before we can make Thanos’ face move, we have to study Josh Brolin’s face. We put him in a 4-D facial scanner. We scan him doing different dialogues, ranges of motion, a database of all the ways his face can move. When Josh is onset he has his motion capture suit on set, he’s interacting with all the other cast members, and he’s also wearing a motion capture helmet and markers. Using these markers along with a database of how his face moves, now we have a realistic Josh face, which we needed to put on Thanos. So we built a machine learning system that understands Josh’s face movements and anatomy, and we teach the system how to transfer Josh to Thanos’ face. And in the end, it’s almost as if Josh is wearing a digital Thanos mask.”

      But, as Darren explained, it's so much more. “This technology is not just digital creatures, it’s digital humans too.” Some examples that Digital Domain has created included the Coachella Hologram of Tupac, or aging Brad Pitt as Benjamin Button.

      So what's next?

    • Darren then shared the next iteration of this amazing technology. Meet DigiDoug:

      “The problem is creating a realistic thanks or Benjamin Button is very difficult. It took teams of people and two companies several years to do. So that’s why up until now you’ve only seen this technology in big Hollywood blockbusters or commercial projects. But this is all about to change. Just about a year ago our team embarked on creating the most realistic digital real-time system. For Thanos, it took us about a month between Josh’s performance to getting it onscreen. But what if we could do this instantly? That is where our journey begins. Enter Doug Robol, our head of software and chief guinea pig, who will famously do anything for science. What we wanted to do with Doug is capture all of the data to create a realistic, real-time digital face. Why the face? It’s the hard part. It didn’t take much effort to create a realistic thanks. But as humans we’re intimately familiar with human faces, able to detect instantaneously if a face is real or fake. So making a digital human can be broken down into two steps: the appearance, and how it moves. To create digiDoug, we started out at the Light Stage X scanner at University of Southern California. They can scan your face down to the pore-level of detail, to see what it looks like in any kind of lighting condition. We can measuring how your blood flow changes under your skin. And we can put that all together into a real-time rendering engine, we get DigiDoug. It doesn’t take hours to generate every frame like in films: it takes milliseconds. These systems together allow us to create images we didn’t think were possible at unbelievable speeds.”

      “We rebuilt the system from the ground up with lots and lots of footage of Doug. Tons and tons of data went into creating the system…different from your iPhone, this system really captures the emotion of the face. And this has been made possible by machine learning. It’s truly revolutionizing everything we do. Computations that used to take minutes or hours to do can now be done instantaneously.”

      “There’s four elements we’re missing. Why are we doing all of this? Well, obviously this new digital realtime technology will be tremendously beneficial for filmmakers. Imagine seeing Josh’s performances live onset. Or actors being able to adjust their performances. Because it’s instantaneous, characters can show up live on TV. Thanos could appear live at Comic-con, interacting with his fans and audiences. And we can bring these characters and digital humans to a wide variety of media, like AR or VR! Imagine a live performance by your favorite musician in your living room. And this isn’t just a thing of the future: there are people working on bringing artificial digital humans to you.”

      Darren then discussed the rise of virtual influencers, like Lil Miquela pictured above, or Hatsune Miku:

      “In the future, when you look at a celebrity, you may not be able to tell if they aren’t real. And these digital versions can be owned and operated like companies. This will change the landscape for influencers. This technology is much bigger than film and TV: soon it will permeate every aspect of your life. Imagine not just talking to Siri or Alexa, but being able to see them, read their facial expressions, see if they understood your question or listened to your conversation. This technology can be used to add a digital face to your conversations every day, to make that experience much more personal. At this moment, our team is using this system to generate massive amounts of data to power realistic assistants. In the next few years it’s likely you’ll interact with a digital human.”

      Darren then shifted over to a new aspect of the technology: “I’m also here to talk about some of the ethical consequences of what you’re starting to see.”

    • Darren then discussed the issue of deep fakes:

      “There’s a whole new realm of technology out there called GANS. And they are able to instantaneously create 2D images of artificial people without having to go through the process of building them up in 3D. So let’s get back to the experts in faces. All of these people are fake. None of them have ever existed. They were generated on a site called ThisPersonDoesNotExist. They created a GAN to synthesize new people, creating a database of new people from images shared online. This technology is not perfect, and there are still flaws to it. They can’t create new expressions.”

      Darren then showcased a video demonstrating the technology in real time:

      “This material was all shot on an iPhone. We created a GAN to instantaneously convert Vicki’s face into mine, with her performance and her emotions. The system didn’t need any artist inputs, just the images it’s trained on. But here’s where it’s scary. This was one of our first versions done a year ago, and the results are getting better every single day. So soon we may be inundated by fake videos and it will get harder and harder to tell what’s really real.”

      “I think it’s important for everyone out there to understand the kinds of things that may soon be possible with this technology. Imagine politicians being able to create emotive artificial videos of themselves targeted directly to you based on your online profile with possibly dubious facts in there. Millions of these could be generated, and only seen by the person targeted. There are many more possible examples out there. And unfortunately there’s no easy solution.”

      “These technologies are developing at breakneck speed, and it’s not going to stop anytime soon. There once was a time when we all believed the camera didn’t lie, that what was in an image truly happened. But that all changed with the advent of photoshop and other digital imaging tools. As a society, we no longer trust photos, we know they may be manipulated in some way. And just as we now know that about photos, we need to start to think that way about videos.”

      Darren finished with:

      “This technology will enhance our entertainment. It will also transform the way we communicate with our devices, bringing a more personal touch to our technology. But there are definitely negative aspects for this technology, and questionable uses for it. But the way we stop it is by being more vigilant about what we consume, and the knowledge of what’s possible out there.”