ASVspoof: every edition from 2015 to ASVspoof 5, its tasks and metrics

This page sits in the audio corner of the deepfake detection hub. It is the edition-by-edition companion to the voice deepfake benchmarks compared page, which ranks the datasets; this one follows the challenge that produced most of them.

01What is ASVspoof and who runs it?

ASVspoof stands for automatic speaker verification spoofing and countermeasures. The series began as a defence of speaker verification systems against four attack families named in the 2021 journal paper: impersonation, voice conversion, text-to-speech synthesis and replay (arXiv 2210.02437). Over five editions the emphasis moved from protecting a speaker verification system to detecting synthetic speech on its own, which is why the series now belongs on a deepfake detection site.

The organisers are a standing academic group. The asvspoof.org homepage lists thirteen people from EURECOM in France, the National Institute of Informatics in Japan, the University of Eastern Finland, Carnegie Mellon University and partners in Hong Kong, Singapore, India and the United States, and names the Academy of Finland, the French ANR, the Japan Science and Technology Agency and Singapore's A*STAR as funders. Data for every edition is distributed from the same site under a download agreement.

02The five editions side by side

Team counts come from each edition's summary paper: Wu et al. for 2015, Kinnunen et al. for 2017, Todisco et al. for 2019 (arXiv 1904.05441), Liu et al. for 2021 and Wang et al. for ASVspoof 5 (arXiv 2408.08739). The 2021 results are the figures recorded for the deepfake audio detection page from the same journal paper.

03Which attacks did each edition test?

ASVspoof 2015 built ten spoofing algorithms on one clean speech corpus. S1 to S5 were known attacks, present in the training and development sets; S6 to S10 appeared only in the evaluation set. The 2017 edition dropped synthesis entirely and tested replay: recordings of RedDots utterances played back through consumer hardware and re-recorded. The replay data raised the equal error rate of a baseline speaker verification system from 1.76 percent to 31.46 percent, which is the measure of how dangerous the attack was before any countermeasure ran.

ASVspoof 2019 was the first edition to cover synthesis, voice conversion and replay in one challenge, split into a logical access scenario with attacks from the latest neural acoustic and waveform models and a physical access scenario with replay generated by controlled simulation (arXiv 1911.01601). The 2021 edition kept LA and PA, added codec and transmission effects to LA, moved PA into real rooms, and introduced the Deepfake task: compressed manipulated speech posted online, with no speaker verification system in the loop. ASVspoof 5 crowdsourced both the speech and the attacks. Its source data comes from the English partition of Multilingual LibriSpeech, with more than four thousand speakers, and its 32 attacks include adversarial attacks built with the Malafide and Malacopula methods, which the organisers describe as a first for the series.

04How did the primary metric change from EER to t-DCF to minDCF?

The equal error rate was the only metric in 2015 and 2017. It is the point where the miss rate on bona fide speech equals the false alarm rate on spoofed speech, and it needs no application assumptions, which suited a young field. Its weakness is the one this site's metrics page describes for every detector: a single operating point hides the cost of each kind of error.

The 2019 edition adopted the tandem detection cost function, t-DCF, as its primary metric, with EER kept as a secondary measure. The t-DCF scores the countermeasure and a fixed speaker verification system together, so it reflects what a spoofing attack does to the system the countermeasure protects. ASVspoof 2021 kept t-DCF for LA and PA, but its Deepfake task had no speaker verification system, so that task was scored by EER alone.

ASVspoof 5 changed again. Its Phase 2 evaluation plan states that Track 1 "builds upon the detection cost function" rather than the EER of past editions, and lists the Track 1 metrics as minDCF (primary), actDCF, Cllr and EER. The minDCF weights a missed bona fide utterance at cost 1 and an undetected spoof at cost 10, with a spoof prior of 0.05, and takes the minimum over thresholds; actDCF uses the threshold a calibrated system would choose, and Cllr measures calibration. Track 2 uses the architecture-agnostic a-DCF, which charges a miss and two kinds of false alarm, from non-target speakers and from spoofs. The summary paper says the organisers adopted minDCF following the practice of the NIST speaker recognition evaluations.

05What were the best published results per edition?

The strongest 2015 system reached an average EER of 1.211 percent, but that average hid a split: 0.408 percent on the five known attacks against 2.013 percent on the five unknown ones. The 2017 replay challenge was harder, with a best EER of 6.73 percent against a baseline of 24.77 percent and an average of 25.91 percent across primary submissions. In 2019 the best LA system, T05, reached a min t-DCF of 0.0069 and an EER of 0.22 percent; 27 of 48 LA teams beat the stronger baseline. The best PA system, T28, reached 0.0096 and 0.39 percent.

The 2021 figures were an order of magnitude worse because the data was. The best LA system scored a min t-DCF of 0.2176 with an EER of 1.32 percent, the best PA system 0.6824, and the best Deepfake task system an EER of 15.64 percent against baselines of 22.38 and 23.48 percent. ASVspoof 5 Track 1 split entries by condition. In the closed condition, limited to the challenge training data, the best team T32 reached a minDCF of 0.2436 with an EER of 8.61 percent, while the two baselines scored above 0.7 minDCF and above 29 percent EER. In the open condition, where external pre-trained models were allowed, T45 reached a minDCF of 0.0750 and an EER of 2.59 percent. In Track 2 the same team led both conditions with a min a-DCF of 0.2814 closed and 0.0756 open.

06Why do the error rates rise across editions?

No two editions share a test set, so a bar for 2021 cannot be compared with a bar for 2015 as if one detector had been tested twice. The 2015 data was clean studio speech with ten attacks; the 2021 Deepfake task used compressed speech from several source databases; ASVspoof 5 used crowdsourced recordings and attacks tuned to fool both verification and countermeasure systems. The 2021 journal paper puts the pattern plainly: detection solutions "lack generalization across different source datasets", and the ASVspoof 5 paper reports that attacks "significantly compromise the baseline systems" that had worked on earlier editions.

The same gap appears in every modality on this site. Face detectors trained on one dataset drop sharply on another, and the effect of video compression on detectors is the audio codec story told in pixels. No single benchmark score predicts how a detector performs on unseen manipulations, and the ASVspoof series is the clearest ten-year record of that rule in audio.

07Where the series stands after ASVspoof 5

ASVspoof 5 ran in two phases. Phase 1, from July 2023 to January 2024, collected attacks from external data contributors who were given a generation protocol and surrogate verification and countermeasure systems to test against. Phase 2 opened on 20 May 2024, took evaluation scores on 21 to 23 July 2024, and ended with the workshop at Interspeech 2024. Both the Phase 1 and Phase 2 evaluation plans and the organisers' summary paper are published on asvspoof.org.

Two of ASVspoof 5's design choices matter beyond audio. The open and closed conditions separate what a model learned from challenge data from what it inherited from a large pre-trained front end, a distinction that face benchmarks rarely make explicit. And the summary paper notes that the top systems in both conditions had actDCF values close to 1.0 because their scores were not calibrated as likelihood ratios: a system can rank utterances well and still be unusable at a fixed threshold. The face anti-spoofing page shows the biometric community reaching the same conclusion about fixed operating points, and the face recognition system components page shows where a threshold sits in a deployed pipeline.