Audio deepfakes in the wild: the 37.9 hour found recording test

This page is the audio counterpart of the in-the-wild strand of the deepfake detection hub. The ASVspoof and in-the-wild audio sets page lists the benchmarks; this one reads the one paper that measured the lab-to-world gap for speech, table by table.

01What is in the In-the-Wild set?

The set holds 37.9 hours of audio: 17.2 hours of deepfakes and 20.7 hours of genuine speech from 58 English-speaking politicians and celebrities, present and past. The authors built the fake half by segmenting 219 publicly available video and audio files "that explicitly advertise audio deepfakes", material whose absurd content makes the label certain, and then collected genuine speech from the same speakers, matching style, emotion, duration and background noise where they could (arXiv 2203.16263). Clips average 4.3 seconds; per speaker there are about 23 minutes of genuine and 18 minutes of spoofed audio.

Two properties separate it from ASVspoof. The deepfakes were made by unknown people with unknown tools for an audience, not by the benchmark's organisers from a known corpus; and the genuine recordings are podcasts, speeches and interviews with their own rooms and microphones, not studio speech. The set is public, and every ASVspoof split, by contrast, derives from the VCTK corpus of professional speakers recorded in a studio.

02How were the detectors trained and scored?

The paper re-implements twelve architectures from the literature: LCNN and its attention and LSTM variants, an LSTM, MesoNet and MesoInception, ResNet18, a Transformer, CRNNSpoof, RawNet2, RawPC and RawGAT-ST. Eight take a spectrogram, computed as constant-Q (cqtspec), log (logspec) or mel-scaled (melspec); four read the raw waveform. Each is trained on the ASVspoof 2019 logical access training and development data, with either a fixed four-second input or the full clip, and each configuration is run three times with random initialisation and averaged.

Scoring follows the ASVspoof convention on the ASVspoof evaluation set, with EER and the tandem detection cost function, and uses EER alone on the In-the-Wild set, because "t-DCF scores require the false alarm and miss costs" that only ASVspoof defines. EER is the point where the miss rate equals the false alarm rate; 50 percent is random guessing. The same models therefore carry two EERs, one per set, and the gap between them is the finding.

03How far did performance fall?

Table 1 of the paper gives every configuration. The rows below are the best spectrogram and raw-waveform configurations and a spread of others, EER in percent.

The authors summarise the right-hand column as EER values that "deteriorate by about 200 to 1000 percent", adding that "often, the models do not perform better than random guessing". The raw-waveform models hold up best on both sets, "obtaining up to 1.2% EER on ASVspoof and 33.9% EER on the 'in-the-wild' dataset"; the spectrogram models reach 6.3 and 37.4 percent. Several configurations exceed 50 percent, which means they rank fakes as more genuine than the genuine speech, and the LCNN with log spectrograms and four-second input reaches 91.1 percent. Averaged over all twelve models, EER is 9.85 percent on ASVspoof and 60.10 percent on In-the-Wild with full-length input, and 18.89 and 67.25 percent with four-second input.

04Does more ASVspoof data help?

No. The authors retrained their best In-the-Wild configuration, RawNet2 with four-second input, on all three ASVspoof 2019 splits, training, development and evaluation, and tested it again: 33.1 ± 0.2 percent EER, "no improvement over training with only the 'train' and 'dev' data". Their explanation is that "all splits of ASVspoof are fundamentally based on the same dataset, VCTK", so the evaluation split adds attack algorithms and speakers but not a new recording world. More of the same benchmark does not buy generalisation.

05What else did the paper find?

Two findings concern recipe rather than data. Replacing mel spectrograms with constant-Q spectrograms "increases the average performance by 37%, all other factors constant", and log spectrograms also beat mel; the authors recommend cqtspec or logspec features. Truncating input to four seconds hurts every model: average ASVspoof EER is 18.89 percent with four-second clips and 9.85 percent with full clips, so papers that fix a four-second window give up accuracy they did not need to.

Neither recipe closes the gap. The conclusion states that "performance drops sharply, and some models degenerate to random guessing", and offers the reading that the field "has tailored its detection models too closely to the prevailing benchmark". The ASVspoof 2021 organisers reached a compatible verdict from inside the benchmark: their systems "lack generalization across different source datasets" (arXiv 2210.02437).

06Do later in-the-wild tests agree?

Deepfake-Eval-2024 repeated the exercise three years later on 56.5 hours of audio collected from social media and detection-platform users in 2024, cut into four-second segments for the open-source models. Its Table 3 reports AASIST at 0.43 AUC on the new set against 1.00 on the ASVspoof 2019 LA evaluation set it was built for, RawNet2 at 0.53 against 0.99 and P3 at 0.58 against 1.00; averaged, open-source audio AUC fell by 48 percent (arXiv 2503.02857). Fine-tuning on 60 percent of the set lifted them to 0.91, 0.88 and 0.92 AUC on the rest, an 80.6 percent gain, and the best of eight anonymised commercial audio models scored 0.93 AUC and 0.89 accuracy. Even fine-tuned, the models were 7.21 percent less accurate on non-English audio and 35.39 percent worse on silent clips.

The two studies use different metrics, EER and AUC, and different sets, so their numbers do not line up, but they agree on direction and size: a detector near perfect on ASVspoof is near chance on found speech until it has seen found speech. The face video equivalent, with the same shape, is on the circulating face deepfakes page, and the face benchmark protocols compared page explains why even curated face sets disagree with one another.

07What the numbers mean for a voice check

A published EER of around 1 percent describes a detector on the benchmark it was trained for, with the benchmark's speakers, codecs and generators. On a recording from the internet, the best 2022 configuration was wrong on one in three decisions at its equal error point, and the best 2024 open-source model scored 0.58 AUC. The human study on deepfake videos found people and a model similarly accurate on faces; no equivalent large study exists for voice, so there is no evidence that a listener does better. A voice in a live call reaches the system by the same software routes as an injected face, and the injection attack test gap applies to audio as it does to video: ENISA's recommended countermeasure is a dedicated application that can "prevent the injection of video and audio", and no public benchmark scores either.