Why deepfake detectors fail in the wild: three causes, each measured

This page is part of the deepfake detection hub. The in-the-wild benchmarks page reports what the found-footage sets found; this page takes the drop apart cause by cause, using the pipeline set out in how deepfake detection works.

01The three causes and where each is measured

A detector learns a cue from the fakes it was trained on. The cue disappears when the fake is made differently, when the file is re-encoded, or when the real footage it is compared against changes character. The table names the controlled measurement for each cause.

The first two rows hold one factor fixed. The third cannot be isolated in any of the published sets, because a different real corpus always arrives with different fakes, so its row combines the causes. All figures were read on 9 November 2026 from the papers cited below.

02Cause 1: why does a new generator remove the cue?

Each manipulation method leaves its own trace, and a detector trained on four of them learns those four. FaceShifter is the cleanest test, because it manipulates the same 1,000 FaceForensics++ source videos with a method released after the original four and is distributed at the same c23 compression. In DeepfakeBench Table 3, Xception trained on FaceForensics++ c23 scores 0.9637 on the original fakes and 0.6249 on FaceShifter (arXiv 2307.01426). Nothing changed but the generator, and a third of the AUC went with it.

DF40's authors state the general form of the problem: "The dominated training dataset, FF++, contains out-of-date forgery techniques from the past four years", and most sets "only contain partial types" of the forgery families that exist (arXiv 2406.13495). Their benchmark of 40 techniques reports as its second finding that well-designed state-of-the-art detectors "may not have obvious advantages over the baseline" once the generator changes. The Deepfake-Eval-2024 authors give the same reason for building a found set: existing datasets "use outdated manipulation techniques (e.g., FaceForensics++ and ForgeryNet)". The frame-level AUC table carries the cross-dataset columns for every detector, not only Xception.

03Cause 2: what does platform re-encoding do?

Platforms re-encode every upload. FaceForensics++ anticipated this: "we compress the videos using the H.264 codec, which is widely used by social networks or video-sharing websites", at a constant rate quantisation parameter of 23 for its high-quality release and 40 for low quality (arXiv 1901.08971). Table 1 of that paper holds the fakes fixed and varies only the compression: XceptionNet reaches an accuracy of 99.26 percent on raw video, 95.73 percent at c23 and 81.00 percent at c40.

DeepfakeBench reproduces the effect with AUC: Xception trained on c23 scores 0.9637 on c23 and 0.8261 on c40, a loss of 0.14 with the manipulations unchanged. Compression removes the high-frequency detail many cues live in, which is why the frequency-domain detectors described on the pipeline page exist. People lose the same way: the FaceForensics++ user study of 204 participants fell from 68.69 percent on raw video to 58.73 percent on low quality. Found footage is compressed by default; Deepfake-Eval-2024 reports an average video resolution of 576 by 720 pixels across its 45 hours.

04Cause 3: why does the real footage matter?

A detector learns what "real" looks like as much as what "fake" looks like. The WildDeepfake authors put the problem in their abstract: in the lab sets, "most of the real videos in these datasets are filmed with a few volunteer actors in limited scenes, and the fake videos are crafted by researchers using a few popular deepfake softwares" (arXiv 2101.01456). Deepfake-Eval-2024 says the same of synthetic video sets, which apply "a limited set of manipulation techniques to curated, often highly structured real videos". A detector tuned to interview footage sees a phone recording, a screen capture or a re-upload as unfamiliar, and unfamiliar reads as suspicious.

The measured consequence is the WildDeepfake result: image-level detectors scoring between 0.90 and 0.99 accuracy on FaceForensics++ or DeepFake-TIMIT fell below 0.70 on 707 internet deepfakes, whose real and fake sequences both came from the same found clips. The DFDC column in DeepfakeBench points the same way, with Xception at 0.7077 on footage of paid actors recorded for the challenge, though DFDC also changes the generators and adds 19 perturbations, so that figure mixes all three causes.

05What happens when all three stack?

Deepfake-Eval-2024 is the stacked case: 45 hours of video, 56.5 hours of audio and 1,975 images collected in 2024 from 88 websites, with unknown generators, platform compression and found real content (arXiv 2503.02857). Its abstract reports AUC "decreasing by 50% for video, 48% for audio, and 45% for image models compared to previous benchmarks". Table 3 gives the rows: GenConViT 0.96 on its original test data and 0.63 on the new set; FTCN 0.87 to 0.50; Styleflow 0.95 to 0.51. Two of the three open-source video detectors sit at chance.

Commercial video models, tested anonymously, ranged from 0.64 to 0.79 AUC (Table S10). Fine-tuning on 60 percent of the new set and testing on the remaining 40 percent raised GenConViT to 0.82 and FTCN to 0.71, an average video improvement of 57.6 percent, but the authors conclude that even these "do not yet reach the accuracy of deepfake forensic analysts". The improvement also measures something different: a model that has seen 2024's distribution is no longer the off-the-shelf tool a user downloads.

06What helps, and what does not

Two things that look like fixes are not. A higher in-dataset score does not help: in DeepfakeBench ten of the 15 detectors trained on FaceForensics++ c23 average above 0.93 within domain, while every cross-domain average sits between 0.6551 and 0.7875, so the spread between methods is small next to the drop. A more elaborate architecture does not help by itself either: the DeepfakeBench authors report that naive detectors "register high AUC values that are comparable to more sophisticated algorithms", and DF40 reached the same conclusion against 40 generators.

What the sources show helping is exposure. Training on more generators, which is the DF40 argument; augmenting with compression during training, which DeepfakeBench lists among its standard augmentations; and fine-tuning on the target distribution, which gave Deepfake-Eval-2024 its 0.82. Each buys accuracy on the fakes that exist now. None transfers to next year's generator, which is the quotable caveat this site attaches to every table: no single benchmark score predicts how a detector performs on manipulations it was not trained on.

Two neighbouring problems have their own pages. Whether a live person is at the camera is presentation attack detection, with error rates per attack type rather than AUC, on the PAD standard explained page. And the human version of cause 1, visual tells that a new generator removes, is on the Which Face Is Real game and its studies page.