Audio deepfake datasets: ASVspoof, In-the-Wild, WaveFake and CodecFake

This page belongs to the deepfake detection hub and gives the dataset-level view behind the audio deepfake benchmarks page, which ranks results. The face-video counterpart is the public deepfake datasets comparison, and the generalizable detection page covers the same cross-dataset problem for face video.

01Which audio deepfake datasets are there?

02How do the ASVspoof sets work?

ASVspoof is a challenge series, and each edition ships a database. The 2019 database is built from the VCTK corpus of 107 speakers (46 male, 61 female), split into three speaker-disjoint sets for training, development and evaluation (arXiv 1911.01601). In the logical access scenario the training set has 2,580 bona fide and 22,800 spoofed utterances, the development set 2,548 and 22,296, and the evaluation set 7,355 and 63,882.

Unseen attacks are part of the design. The evaluation partition mixes 2 known attacks with 11 unknown ones, so a system is always scored on generators it was not built from. ASVspoof 2021 added a deepfake task for compressed speech, and the ASVspoof 2021 journal paper reports 54 teams. ASVspoof 5 (2024) moved to crowdsourced data from more than 4,000 speakers and 32 attacks, with A01 to A08 in training, A09 to A16 in development and A17 to A32 in evaluation (arXiv 2408.08739). Adversarial attacks appear for the first time. The ASVspoof page and the ASVspoof 5 page cover the tasks and metrics.

03What did the In-the-Wild set show?

Müller et al. built it because they judged that ASVspoof evaluation audio "share certain properties with the training data", so generalization cannot be assessed on it alone (arXiv 2203.16263). The set has 37.9 hours of English-speaking celebrities and politicians, 17.2 hours fake and 20.7 hours real, from 58 speakers, with an average clip length of 4.3 seconds. The fake clips were cut from 219 public video and audio files that advertise audio deepfakes, and genuine clips from the same speakers were collected from podcasts and speeches.

The authors re-implemented and uniformly evaluated architectures from earlier work. Averaged over models, the equal error rate was 9.85 percent on ASVspoof 2019 evaluation data and 60.10 percent on In-the-Wild for full-length input, as the figure shows. The abstract states the worst case as a performance degradation of "up to one thousand percent". The paper reports that its raw-waveform models, RawGAT-ST and RawNet2, scored as low as 1.2 percent on ASVspoof and 33.9 percent on In-the-Wild. The In-the-Wild page gives the model-level table.

04What is in WaveFake?

WaveFake (NeurIPS 2021 Datasets and Benchmarks) has 117,985 generated audio clips, about 196 hours, built from two public reference sets: LJSpeech, 13,100 clips of about 24 hours read by one female speaker, and the basic5000 corpus of JSUT, a Japanese set of about 6.7 hours (arXiv 2111.02813). The generators are vocoders: MelGAN in two sizes, Full-band and Multi-band MelGAN, HiFi-GAN, WaveGlow and Parallel WaveGAN, plus one full text-to-speech pipeline.

Because the generators recreate the training distributions of the reference sets, clips can be compared one to one across architectures, and the authors also generate novel phrases. The authors' baselines are a Gaussian mixture model and RawNet2, scored by equal error rate; a mixture model trained on FB-MelGAN output gave the lowest average error across the other vocoder sets, 0.062.

05What is CodecFake?

Two datasets use the name, both at Interspeech 2024. Wu, Tseng and Lee re-synthesise the VCTK corpus through 15 neural codec models from six codec types, split into three speaker-disjoint sets, and add test audio from a VALL-E re-implementation (arXiv 2406.07237). Models trained on ASVspoof 2019 had equal error rates "usually higher than 30%" on most codec test subsets; models trained on CodecFake reached 0.53 and 0.4 percent on VALL-E audio. A separate dataset by Lu and colleagues uses seven neural codec methods (arXiv 2406.08112).

06Why do figures differ between sources?

Three discrepancies are worth knowing. The In-the-Wild paper gives 20.7 hours of real audio in its text and 20.8 in a figure caption; 17.2 and 20.7 sum to the stated 37.9. The WaveFake arXiv abstract page says nine sample sets from five architectures, while the paper text says ten from six and lists seven sets from one reference corpus and two from the other; the clip and hour totals on this page come from the paper text. And ASVspoof 2021 results depend on the task: the top team scored an equal error rate of 1.32 percent in logical access and 15.64 percent in the deepfake task, so a quoted ASVspoof score needs its edition and task.

07How should an audio score be read?

Name the dataset, the edition, the task and the metric. Equal error rate measures one threshold; ASVspoof 5 moved its primary metric to minimum detection cost. The equal error rate page defines it. For a test of generalization, prefer data whose generators and recording conditions are absent from training, which is what the unseen attacks and In-the-Wild sets are for.