ASVspoof 2021: the LA, PA and DF tasks, the data and the leading systems

This page is the 2021 entry in the deepfake detection hub and expands the one-row summary on the ASVspoof challenge series page: three tasks, their data, the baselines, the top systems and what changed after the challenge closed.

01What were the three tasks?

The LA task kept the 2019 question, telling bona fide speech from text-to-speech and voice conversion output, but passed every utterance through real telephony systems. The PA task asked systems to detect replayed recordings, this time captured in real rooms rather than simulated ones. The DF task was new: compressed, manipulated speech "posted online", with no speaker verification system in the loop, scored by the countermeasure equal error rate alone (Liu et al., 2023).

LA and PA used the minimum tandem detection cost function (min t-DCF) as the primary metric because an automatic speaker verification system sits behind the countermeasure and the cost of each error depends on both. The DF task "does not involve the use of an ASV system", so its primary metric is the CM EER; the equal error rate page defines that term. EER was reported for LA and PA as a secondary figure. The contrast with video is worth noting: the Deepfake Detection Challenge of 2020 ranked its entries by log loss, a per-video probability score, where ASVspoof ranks by error rates at a chosen operating point.

02How much data did each task hold?

Each task was split into a progress subset, scored during the challenge, and an evaluation subset, scored once at the end; the table below reproduces the counts from Table I of the paper.

The LA speech came from the ASVspoof 2019 LA evaluation database, and both the LA and PA data derive from the VCTK corpus. The DF task added two further sources, the Voice Conversion Challenge 2018 and 2020 databases, which contributed 26 more speakers and conversion systems never used in an earlier ASVspoof edition; the paper states that the full DF evaluation set was "generated using in excess of 100 different spoofing attack algorithms". Training was restricted to the ASVspoof 2019 LA training and development partitions, which contain only clean audio.

03Which codecs and channels did the data pass through?

The LA task defined seven conditions (Table II). C1 was the clean 16 kHz reference at 250 kbps. C2, C4, C5, C6 and C7 passed speech through an Asterisk private branch exchange over VoIP using a-law (8 kHz, 64 kbps), G.722 (16 kHz, 64 kbps), mu-law (8 kHz, 64 kbps), GSM full rate (8 kHz, 13 kbps) and OPUS (16 kHz, variable bitrate 16 kbps). C3 sent calls from a mobile phone over a public switched telephone network in Spain to a mu-law VoIP endpoint, so its intermediate codecs were unknown. Transmission ran within a local network or from France to endpoints in Italy and Singapore.

The DF task used storage codecs instead (Table IV): mp3 at roughly 80 to 120 kbps and 220 to 260 kbps, m4a (advanced audio coding) at about 20 to 32 kbps and 96 to 112 kbps, and ogg Vorbis at about 80 to 96 kbps and 256 to 320 kbps, with a no-codec reference and two transcoding conditions, mp3 to m4a and ogg to m4a. The PA task recorded replays with real loudspeakers in real rooms, described in the paper's Table III, after the 2019 edition had shown over-fitting to simulated attacks.

04What were the four baselines?

All three tasks shared four organiser baselines (Table V). B01 used constant Q cepstral coefficients with a Gaussian mixture model; B02 used linear frequency cepstral coefficients with a GMM; B03 fed LFCC features to a light CNN with an LSTM layer; B04 was RawNet2, an end-to-end network that reads the raw waveform. On the evaluation sets B03 was the strongest baseline for LA (min t-DCF 0.3445, EER 9.26 percent), B01 for PA (0.9434, EER 38.07 percent) and B04 for DF (EER 22.38 percent).

05Who scored best on each task?

The paper's Tables XIV to XVI rank every team by its evaluation-set score. The challenge allowed anonymous participation, so systems are known by team identifiers; entries without an identifier submitted no system description. The table below gives the top five per task.

The paper prints the LA winner's score as 0.2176 in its abstract and Table VIII and as 0.2177 in Table XIV; this site uses 0.2176, the figure the authors quote in prose. On the DF task 18 of 33 ranked systems beat the best baseline, and on LA the top systems sat close to the ASV floor, the t-DCF a perfect countermeasure would leave. On PA the best min t-DCF of 0.6824 is far from that floor, which the abstract attributes to "a lack of robustness to variations between simulated and real acoustic environments".

Table V describes the winning recipes. T23's LA system augmented training data with transmission codecs, took raw waveforms and raw spectrograms as input, ran LightCNN, ResNet and LSTM classifiers and fused them by weighted score averaging; its DF entry used the same design with media codec augmentation. T07, the PA winner, used speed perturbation, log spectrograms and a variational autoencoder with a GMM.

06What did the DF results show about compression and source data?

The DF task separated two effects. Systems coped with compression: the abstract states that "detection solutions offer some resilience to compression effects". They did not cope with new source data: the same sentence continues "they lack generalization across different source datasets", and the paper reports that the best system scored below 1 percent EER on the progress subset while every system exceeded 15 percent on the evaluation subset, where the VCC 2018 and 2020 material appeared. Spoofs built with neural vocoders produced higher EERs than traditional vocoders or waveform concatenation, and ogg-coded trials gave lower EERs than mp3 or m4a.

That pattern repeats on found audio and in face video. The In-the-Wild set of 2022 reports "performance degradation of up to one thousand percent" for detectors moved from lab data to recordings collected online (Müller et al., 2022), and the DF40 face benchmark of 2024 found the same drop when a detector met unseen generators. The practical side appears on the deepfake fraud statistics page: the official bodies that warn about cloned voices publish no detection figures at all.

07What came after the challenge?

Table VIII of the paper collects post-challenge results from published papers. On the LA database a system with a self-supervised front end reached a min t-DCF of 0.2066 and an EER of 0.82 percent, against the challenge best of 0.2176 and 1.32 percent; on the DF database the post-challenge best EER fell to 2.85 percent from 15.64 percent. The authors note that such front ends are pre-trained on external data, which "is not compliant with the ASVspoof training protocol", so the two columns are not a like-for-like ranking. End-to-end models on raw waveforms, with AASIST as the representative example, and data augmentation are the two trends the paper names.

The next edition changed the primary metric again. The ASVspoof 5 evaluation plan makes the detection cost function primary for its stand-alone deepfake track and keeps EER as a reported figure, as the ASVspoof challenge series page describes. No single ASVspoof score predicts how a countermeasure performs on speech from a source it was not trained on; the 2021 DF task is the clearest published demonstration of that.

The next edition changed the task split; the ASVspoof 5 tracks and metrics page records what it kept and what it replaced.