Voice deepfake benchmarks: ASVspoof and the in-the-wild audio sets

This page is the one audio page in the deepfake detection hub. The scores on this site's results page are for face video; the voice figures here are kept separate because the metrics and the test material differ, but the pattern they show is the same.

01How voice deepfake detection is benchmarked

A spoofed utterance is speech that was synthesised from text, converted from another speaker's voice, or replayed through a loudspeaker. A countermeasure, the field's word for a detector, scores each utterance as genuine or spoofed. The benchmark fixes three things: which attacks appear in training, development and evaluation, so that the evaluation set can hold attacks the detector never heard; whether the detector is scored alone or together with an automatic speaker verification system it is meant to protect; and the metric. ASVspoof's evaluation sets have always held unseen attack types, which makes every ASVspoof number a partly cross-attack result, the audio equivalent of the cross-dataset rows on this site.

02The ASVspoof challenges

The series has had editions in 2015, 2017, 2019 and 2021, followed by ASVspoof 5 in 2023 and 2024, organised by groups including EURECOM, the University of Eastern Finland, NII in Japan and others (asvspoof.org). The 2021 edition had three tasks and 54 participating teams: logical access (LA), where synthetic or converted speech is injected into a telephone channel; physical access (PA), where speech is replayed in a room and recorded by a microphone; and a deepfake (DF) task, new in 2021, where compressed synthetic speech is scored without any speaker verification system involved (arXiv 2210.02437). The LA and PA tasks map onto the injection and presentation attacks of the liveness page; the DF task is the audio counterpart of judging a circulated clip.

The 2021 summary paper reports the top systems per task. For LA the best entry scored a minimum t-DCF of 0.2176 and an EER of 1.32%. For PA the best minimum t-DCF was 0.6824. For DF the best entry's EER was 15.64%, against the organisers' baselines B04 at 22.38% and B03 at 23.48%. The paper also cites post-challenge systems, built once the evaluation data were released, reaching a t-DCF of 0.2066 (EER 0.82%) on LA and an EER of 2.85% on DF; those are a different protocol, since the test distribution was no longer unseen.

ASVspoof 5 split the problem into two tracks: stand-alone deepfake detection, scored by a minimum detection cost function as the primary metric with EER and other measures alongside, and spoofing-robust speaker verification, scored by a minimum architecture-agnostic detection cost function (evaluation plan).

03Metrics: EER and t-DCF

The equal error rate is the point on the detector's error curve where the rate of spoofed utterances accepted equals the rate of genuine utterances rejected. It is the audio field's usual single number and the same quantity the metrics page defines for face detectors, with the same caveat: it is one threshold, and a deployment may run at another. The tandem detection cost function (t-DCF) was introduced because a countermeasure in front of a speaker verification system is only as useful as the combined system; it weighs the costs of the two systems' errors together, with assumed costs and a prior on how often attacks occur. The 2021 paper uses t-DCF as the primary metric for LA and PA, where a verification system is present, and EER for DF, where it is not. A t-DCF is not comparable with an EER, and neither is comparable with the AUC values on this site's face results table.

04In-the-wild audio sets

Müller, Czempin, Dieckmann, Froghyar and Böttinger assembled a set of found recordings to test whether lab results hold outside the lab: "37.9 hours of found audio recordings of celebrities and politicians, of which 17.2 hours are deepfakes". Their finding is that "related work performs poorly on such real-world data (performance degradation of up to one thousand percent)" and that "deepfakes are much harder to detect outside the lab than previously thought" (arXiv 2203.16263, Interspeech 2022). Deepfake-Eval-2024 added 56.5 hours of circulated audio from 2024 and reports open-source audio detectors' AUC falling by 48% against their original benchmarks (arXiv 2503.02857). The in the wild page covers that study across all three modalities, and the datasets table lists the audio portion of Deepfake-Eval-2024 with the face sets.

05Face and voice results side by side

The two fields cannot share a table, but they share a shape. In face detection, ten of the fifteen detectors in the shared DeepfakeBench evaluation score above 0.93 frame-level AUC on the FaceForensics++ split they trained on, and all fifteen score between 0.55 and 0.95 on unseen sets. In voice detection, the ASVspoof evaluation sets are built to contain unseen attacks, the best 2021 deepfake-task EER was still 15.64%, and found recordings degrade lab-trained detectors by up to tenfold. In both, the best published numbers come from benchmarks whose test distribution the detector did not see, and both fall further on material that circulated. The results page keeps the face rows with their conditions; the voice figures stay here with theirs.

The 2024 edition is covered on its own page, ASVspoof 5 explained, with its stand-alone detection and spoofing-robust ASV tracks.