Deepfake detection in the wild: what benchmarks built from real circulating fakes found

This page is part of the deepfake detection hub. It is the practical end of the cross-dataset generalisation story: that page shows the drop between two curated sets, this one shows the drop between a curated set and the internet.

01Lab benchmark versus the wild

Every curated dataset makes choices that a circulating fake does not respect. FaceForensics++ was made from 1,000 YouTube videos manipulated with four methods and released at three known compression levels. The Deepfake Detection Challenge set was recorded with paid actors and manipulated with eight methods. In each case the test split shares its generators, its source footage and its compression with the training split, so a detector can learn traces that are specific to the set and still score well on it. A fake found on a social platform has none of those guarantees: it may come from a generator released last month, it has been re-encoded by the platform at least once, it may have been cropped, captioned or screen-recorded, and nobody knows what the original looked like.

Two kinds of benchmark try to measure that. The first collects fakes from the internet after the fact. The second takes a curated set and applies the kinds of damage platforms inflict. Both are described, with counts, on the datasets page; this page is about what they found.

02In-the-wild benchmarks

WildDeepfake (ACM Multimedia 2020) was the first: 707 deepfake videos collected from video-sharing sites, processed into 7,314 face sequences and 1,180,099 face images, split 6,508 train and 806 test, with 3,805 real and 3,509 fake sequences in all (arXiv 2101.01456). Because the clips were found rather than made, the authors do not know which methods produced them.

Deepfake-Eval-2024 (2025) is the largest and most recent: 45 hours of video, 56.5 hours of audio and 1,975 images collected from social media and from users of detection platforms during 2024, from 88 websites in 52 languages (arXiv 2503.02857). It is multimodal, so the same question is asked of face video, voice and still images.

The audio field has its own version. The In-the-Wild audio set (Interspeech 2022) holds 37.9 hours of found recordings of celebrities and politicians, 17.2 hours of them deepfakes (arXiv 2203.16263); the voice deepfake benchmarks page covers it.

03What they found

WildDeepfake's authors ran ten image-level baseline detectors on their set. Detectors that score between 0.90 and 0.99 accuracy on FaceForensics++ (Deepfake) or DeepFake-TIMIT all fell below 0.70 accuracy on WildDeepfake; the authors' own method reached 76.25%.

Deepfake-Eval-2024 reports the drop for the current generation. The paper's Table 3 gives each open-source model's AUC on the benchmark it was built for and on the new set: for video, GenConViT fell from 0.96 to 0.63, and FTCN and Styleflow scored 0.50 and 0.51 on the new set, at chance. Averaged over models, AUC fell by 50% for video, 48% for audio and 45% for images. Commercial video models, tested anonymously, ranged from 0.64 to 0.79 AUC. Fine-tuning GenConViT on part of the new set raised it to 0.82, which the authors report separately because it is a different protocol: a model that has seen the distribution is no longer off the shelf. The paper's summary is that commercial and fine-tuned models do better but "do not yet reach the accuracy of deepfake forensic analysts".

The Deepfake Detection Challenge paper had reported the opposite sign in 2020: its winning model, trained only on the challenge set, kept a ROC-AUC of 0.734 on a small set of real in-the-wild deepfakes the authors collected (arXiv 2006.07397). Both can be true. The 2020 result measured 2020 fakes against a model trained on a large, perturbed, consented set; the 2024 result measured 2024 fakes against models trained on older benchmarks. The results page carries both rows with their conditions, and the DFDC page describes the set behind the first.

04Why the gap exists

Three causes stack. The generator changes: DF40's authors describe FaceForensics++ as containing "out-of-date forgery techniques from the past four years", and a detector trained on them has never seen the artefacts of a diffusion-based swap. The compression changes: platforms re-encode every upload, and the FaceForensics++ paper itself shows the same detector falling from 99.26% to 81.00% accuracy between its uncompressed and heavily compressed releases on the same fakes. And the real videos change too: curated sets use clean interview footage, while circulating clips are phone recordings, screen captures and re-uploads, so a detector tuned to one kind of "real" raises false alarms on another. The cross-dataset generalisation page quantifies the first cause with the published tables.

05Statistics worth quoting, and their sources

Figures about deepfakes travel fast and lose their sources on the way. The ones below are the only ones this page will quote, each with where it comes from.

• Open-source detectors' AUC fell by 50% for video, 48% for audio and 45% for images between their original benchmarks and Deepfake-Eval-2024 (Chandra et al., 2025, abstract).
• The best off-the-shelf open-source video detector scored 0.63 AUC on Deepfake-Eval-2024; the best commercial one 0.79 (same paper, Tables 3 and 5).
• Image detectors scoring 0.90 to 0.99 accuracy on FaceForensics++ or DeepFake-TIMIT scored below 0.70 on WildDeepfake (Zi et al., 2020).
• Audio detection methods degraded by "up to one thousand percent" on found recordings compared with lab data (Müller et al., 2022, abstract).
• People shown StyleGAN2 faces classified them as real or synthetic with 48.2% accuracy, against 50% chance (Nightingale and Farid, 2022, experiment 1).

Vendor surveys, market sizes and counts of "deepfakes online" are not listed here because none of them is reproducible from a primary source.

06What this means for checking a video

A detector score on a clip from the internet is a weak piece of evidence, and the published in-the-wild figures say how weak: a coin toss for several open-source models, four in five for the best commercial one in 2024. That is why the checking a face video page puts a detector score last, after provenance, context and corroboration, and why the human studies matter: when the tools and the eye are both near chance on synthetic faces, the question "where did this come from?" does more work than "does this look real?".