It sounded to many media just too spectacularly tempting to ignore it: The head of a British company allegedly got a call from the CEO of the German parent company - with the instruction to immediately transfer 220,000 euros to a specific account. Because the Briton believed he recognized the German's voice and accent, he made the payment. But the voice was fake, generated by a deep-fake software that had been trained with sound recordings of the CEO. The money had landed with scammers.
This is - shortened - the story, which went in June by German media, two weeks ago in the "Wall Street Journal" appeared and now appears again in the German media. The only source is the insurance of the unnamed group, Euler Hermes. However, there is no evidence that the story is true.
"New Scam: First Fake President Case with Impersonation by AI Software," writes Euler Hermes in his press release. Fake President is called the scam, where the perpetrators pretend to be bosses, AI stands for artificial intelligence, in this case deep learning, the technical basis for deep fakes.
But when asked, how do you know that such a software and no voice imitator has been used, a spokeswoman replies to SPIEGEL by email: "We do not know with 100% certainty. Theoretically, it could have been a human voice imitator But we do not assume that there are some clues (but no evidence). "
In turn, the clues she mentions have no technical relevance; they are by no means interpreted as evidence of a deepfake. The supposedly "first case" can therefore at best be termed a "possible case". Which is quite symptomatic of the debate about deepfakes and the resulting risks.
The bleak scenario that politicians, IT security companies, researchers, the media, and even intelligence agencies are warning about looks like this: It's becoming increasingly easy to use deepfakes to put any face into any video and mimic any part of a voice without it being obvious , Anyone who synthesizes the facial expressions and voices of heads of government, CEOs, or other powerful people and lets them say any sentences has potentially very effective tools for disinformation, slander, fraud, and blackmail.
The reality looks - September 2019 - less gloomy. The technology is evolving rapidly. But deepfakes today are either easy to make or (at least to some extent) convincing. With the Chinese app Zao, for example, users can cut themselves into familiar film scenes via selfie, but that does not look realistic. For comparatively high-quality so-called face swaps, ie the exchange of faces or at least the mouth part in a video, one needs special hardware and software, at least half a day, but better several days time and experience in dealing with training data and best of all also the fine tuning of Machine-learning models. This video tutorial, for example, gives an idea of the amount of work required.
But even after a successful face swap, the new face still has the old voice. A case in which images and soundtrack were created convincingly by a Deepfake software, has not been previously known. The biggest damage that deepfakes have done so far is likely to be the humiliation of people whose faces are mounted in porn videos - regardless of the quality of the fake.
At the same time, sizeable sums are being spent on the development of new tools that will detect counterfeits even when the human eye or hearing is overwhelmed. Facebook, Microsoft and several other American companies and universities, for example, have just won a contest and a total of $ 10 million in prize money. Darpa, the Department of Defense research agency, has already distributed $ 68 million, half of its four-year funding program, including Hany Farid, a professor at Berkeley and a pioneer in deep-fake detection.
The facial expression method
Together with colleagues, Farid has found a way to extract the typical facial muscle and head movements of a person as they speak from a video and to form a model of it. The movements are so minimal and individual that today's deep-fake algorithms can not reproduce them. When comparing Farid's tool therefore recognizes a fake video with high reliability, in which an entire face or even the lip area was exchanged to put the people in the video any words in the mouth.
Farid wants to make it available to journalists on a website in December, including the models of all candidates for the US presidential election 2020. Deepfakes of other celebrities, the technology accordingly, initially not recognize, the tool is designed specifically for the US election and would Therefore, not a panacea even if it would always be superior to the deepfakes generators. Farid says in an e-mail to SPIEGEL: "This technique is not meant to analyze all videos on YouTube or Facebook, but is part of a larger toolbox to help journalists verify a story."
The "mouthnet" method
Other researchers and companies are working on general solutions that can be applied to any video. Mouthnet is an example of such a system, developed by Matt Price, researcher at the IT security company ZeroFox. Mouthnet analyzes the mouthparts in videos and extracts clearly visible as well as inconspicuous features from the individual pictures. A common gross error of deepfake generators is the representation of teeth as a contiguous, even series. More subtle, on the other hand, are certain digital artifacts that are created when creating a fake video. "Mouthnet realizes that some pixels are not from a camera," says Matt Price. Teeth will soon look more realistic, he suspects, but the telltale pixels "will not disappear that fast."
So far, however, recognizes his model only about every second Deepfake video and considers every fourth real video for a fake - for everyday use is still unfit. On the other hand, the FaceForensics (++) algorithm developed at the Technical University of Munich recognizes just under four in five Deepfake videos.
The mouse method
However, the idea that George Williams presented at the IT security conference Black Hat in Las Vegas in August is a vision of the future. Williams works for hardware manufacturer GSI Technology of California, which among other things develops microprocessors for deep learning applications. Together with neurobiologist Jonathan Saunders and data scientist Alexander Comerford, he claims: Mice can tell people if a voice is real or mimicked by a system such as Google's Tacotron 2.
"Mice have a similar hearing system to humans," says Williams. "But you do not have to see any sense in the sounds you play to them, so you recognize artifacts that point to synthesized voices better than we humans do." It is possible to teach mice an 80% chance of detecting counterfeit images in 18 weeks, says Williams.
How well do people recognize deepfakes ...
... if they know that they get to see some?
Depending on the test situation, people will recognize fake videos or audio recordings differently. If you want to say which of two shots is a deepfake, on average 88% of them are correct. This is the result of a hitherto unpublished study by George Williams of GSI Technology.
... if they are unprepared?
In a more realistic scenario, it looks different. According to a study by Professor Matthias Nießner of the Technical University of Munich, inexperienced people recognize highly compressed videos, as they are typical in social media, on average "only with a probability of just over 50 percent correct".
Of course, the goal is not to train millions of mice and distribute them to millions of people, but to replicate their ability digitally as a pattern in software, so to speak. But much research work is needed before that can happen: "We're talking years," says Saunders.
The marking method
Firms like Truepic and projects like ProofMode are in the very beginning, in the cameras. Truepic has developed an app that marks each photo and video with a kind of digital watermark on capture. It consists of data from the image sensor as well as various metadata such as location and time of a recording, is cryptographically signed, encrypted in a database or Blockchain transmitted and stored. Each shot photo that appears on the Internet can be checked for its origin and integrity by the watermark.
The drawback of this technique: It has to be widespread and accepted, for example by integration into the camera software of the iPhone, so that people become suspicious of images and videos without watermarks and take a closer look.
Who takes responsibility?
In addition to finding the right detection technology, this is the next unresolved question on dealing with deep fakes: Ultimately, should anyone be responsible for the detection and if so, who? Hardware manufacturers? The operators of social networks and other online platforms? Every single internet user? For various reasons, all three approaches are difficult to imagine.
Even if all hardware manufacturers were legally obliged by all governments or voluntarily agreed to integrate something like Truepic, there would be billions of old, non-watermarked devices in the world. This also makes it clear that never all people could be forced to create exclusively manipulation-protected content or to check everything for authenticity, what they encounter on the Internet. At best, it would be conceivable that the media would commit themselves to watermarking their own content and that only verified content in the coverage should be considered authentic. But those who do not trust the media today will not do that either.
Still remain the service providers. "I imagine these techniques are used by the Facebooks, Twitters and YouTubes of this world," says Hany Farid, and he's not alone in that opinion. At least the scaling problem would be solved: a deepfake detector for millions of users.
There would be another monitoring layer in the network, because a deepfake detection at the platform level would be nothing more than an additional upload filter or a downstream filter. On Facebook, Twitter, YouTube and other sites would add to the algorithms, which should automatically sort out terrorist content, documented child abuse and copyright infringement. And like the others, he can sometimes be wrong, which the users would have to remember somehow.
Jonathan Saunders, the neurobiologist with the mice, therefore does not believe in a technical, but in a social response to the deep-fake phenomenon: "The lesson that we learned through Photoshop is the model." There were a number of fake photos that did a lot of damage, but we adjusted our expectations. " Once people have encountered a number of compelling deep-fakes, "they will stop believing everything."
