ASVspoof 5: its two tracks, its metrics and what changed from 2021

This page is part of the deepfake detection hub and goes one level deeper than the voice deepfake benchmark overview, which places ASVspoof beside the found-recording sets. Here the subject is the fifth edition alone: its tracks, its database, its metrics and its results.

01What is ASVspoof 5?

ASVspoof is a challenge series, with editions in 2015, 2017, 2019 and 2021, in which participants submit scores for a shared evaluation set and the organisers rank them. The fifth edition was "previously-biennial" no longer: it ran in two phases, with Phase 1 registration open from 1 July 2023 to collect crowdsourced attacks from external contributors, and Phase 2 registration from 20 May 2024 with evaluation scores due between 21 and 23 July 2024 (asvspoof.org). Results were presented at the ASVspoof Workshop at Interspeech 2024 on 31 August 2024, and the summary paper records results "submitted by 54 challenge participants" (arXiv 2408.08739).

The challenge site lists thirteen organisers from institutions in Europe, Asia and North America, and the summary paper thanks data contributors from fourteen institutions who built the attacks. That structure is new: in earlier editions the organisers made the attacks themselves.

02What are the two tracks?

Track 1 is "robust speech deepfake (DF) detection in a stand-alone setting without speaker verification, similar in spirit to the DF task in ASVspoof 2021", as the Phase 2 evaluation plan puts it (evaluation plan). A submission assigns one bona fide versus spoof score to each utterance. The scenario the summary paper describes is an attacker who clones a victim's voice and posts the result, so detection has to survive conventional codecs such as mp3 and neural codecs.

Track 2 is spoofing-robust automatic speaker verification (SASV): the system must accept the target speaker and reject both other speakers and spoofed speech, in a telephony scenario where converted or synthetic audio is injected into the channel with no acoustic propagation. Participants could submit a single SASV score per trial or separate ASV and countermeasure scores to be fused. Both tracks had a closed condition, restricted to the challenge training and development sets, and, for the first time, an open condition allowing external data and pretrained speech foundation models.

03How was the database built?

Earlier editions drew bona fide speech from VCTK, which the summary paper describes as "around 100 speakers' data recorded in an anechoic chamber". ASVspoof 5 instead used the English partition of Multilingual LibriSpeech, "more than 4k speakers, recorded with diverse devices". The organisers split those speakers into three disjoint partitions: one group of contributors built text-to-speech systems on partition A; a second group built text-to-speech and voice conversion systems on partition B and tuned them against surrogate ASV and countermeasure systems; the tuned systems then cloned the voices in partition C for the development and evaluation sets. Spoofing attacks were therefore "optimised to fool not only ASV but also CM surrogate sub-systems", which earlier editions did not do.

The paper's Table 1 gives the counts: the training set holds 18,797 bona fide and 163,560 spoofed utterances from 400 speakers with 8 attacks; the development set 31,334 and 109,616 with 8 attacks; the Track 1 evaluation set 138,688 bona fide and 542,086 spoofed utterances from 737 speakers with 16 attacks. The 32 attacks A01 to A32 are disjoint across sets, with A17 to A32 reserved for evaluation. Among the evaluation attacks are adversarial variants built with the Malafide and Malacopula filters, incorporated "for the first time", and zero-shot cloning with two pretrained text-to-speech systems. Evaluation utterances are also passed through codec conditions C00 to C11, seven at 16 kHz including a neural codec and four narrow-band conditions downsampled to 8 kHz.

04What do the Track 1 metrics mean?

Past editions ranked countermeasures by equal error rate. The evaluation plan states the change directly: "Different to past ASVspoof challenge editions for which the equal error rate (EER) was used as the primary metric for the comparison of spoofing countermeasures (CMs), Track 1 builds upon the detection cost function (DCF)". The normalised DCF is a weighted sum of the miss rate (bona fide speech rejected) and the false alarm rate (spoofs accepted), with the weight set from the costs and a prior: Cmiss of 1, Cfa of 10 and a spoof prior of 0.05, which the paper says "gives β ≈ 1.90". The assumption is that spoofs are rare but costly to miss.

Four numbers are reported per system. minDCF is the cost at the best threshold chosen with knowledge of the labels, and is the primary metric. actDCF is the cost at a fixed threshold set from the costs and priors, meaningful "only when the scores can be interpreted as calibrated LLRs". Cllr, the cost of log-likelihood ratios, scores calibration directly, lower being better. EER is kept as a secondary figure. The detector metric definitions page gives the face-detection vocabulary these map onto, and explains what EER does and does not say.

05What does Track 2's min a-DCF measure?

Track 2 moved from the EER-based scoring of the first SASV challenge to a cost: the architecture-agnostic detection cost function (a-DCF) weighs three error rates at one SASV threshold, the target speaker miss rate, the false acceptance of non-target speakers and the false acceptance of spoofs. The priors are 0.9405 for targets, 0.0095 for non-targets and 0.05 for spoofs, with Cmiss of 1 and both false alarm costs of 10, which gives weights of about 1.58 and 0.84 in the paper's notation. The primary metric is its minimum over thresholds, min a-DCF.

Submissions that exposed separate ASV and countermeasure scores were also scored with the ASV-constrained minimum tandem DCF, the primary metric since 2019, computed with the organisers' common ASV system, and with the tandem EER, the single operating point at which the miss rate and both false alarm rates are equal.

06What were the results?

Track 1's two baselines, RawNet2 (B01) and AASIST (B02), both raw-waveform models that had worked on earlier editions, "achieved minDCF higher than 0.7 and EERs higher than 29%": 0.8266 and 36.04 percent for B01, 0.7106 and 29.12 percent for B02. The best closed-condition submission, T32, scored a minDCF of 0.2436 with an EER of 8.61 percent; the top five closed submissions all had minDCF below 0.5 and EER below 15 percent, "around 50% relative improvement over the baselines". In the open condition the best submission, T45, reached a minDCF of 0.0750 and an EER of 2.59 percent, and "most of the well-performing submissions use features extracted by pre-trained self-supervised learning (SSL) models, e.g., wav2vec 2.0".

Track 2 drew fewer submissions, which the organisers attribute to the harder task. The organisers' own ASV without a countermeasure (REF) scored a min a-DCF of 0.6869; the fused baseline B03 scored 0.6806, close to REF, which "indicates that the CM sub-system in B03 does not provide useful information". T45 led both conditions, with 0.2814 closed and 0.0756 open. A caveat runs through both tables: "the top systems in both conditions obtained actDCF values close or equal to 1.0", because their scores were normalised to between 0 and 1 rather than calibrated as likelihood ratios, so the fixed-threshold cost was as bad as accepting everything even where the minimum cost was excellent.

07What changed from ASVspoof 2021?

Five things. The primary metric moved from EER to minDCF for stand-alone detection, and from a tandem DCF to a-DCF for verification. The source speech moved from about 100 studio-recorded VCTK speakers to more than 4,000 crowdsourced ones. Attacks were tuned against surrogate detectors and, for the first time, made adversarial. An open condition admitted external data and foundation models, and the best open systems used them. And the logical access task of the 2021 edition became a full SASV track. The page on every ASVspoof edition lines the five up.

What did not change is the gap that the in-the-wild face benchmarks show for video: a challenge evaluation set, however hard, is a set its organisers built. No single benchmark score predicts how a detector performs on recordings its test set did not contain, and the ASVspoof 5 baselines, which had scored well on 2021 data and scored minDCF above 0.7 on 2024 data, are the series' own demonstration of that.