Equal error rate (EER): what it measures and where benchmarks report it

This page is part of the deepfake detection hub and sits with the other metric definitions on the metrics page. It explains what EER fixes that AUC leaves open, and why the voice benchmarks chose it.

01What is the equal error rate?

A binary detector produces a score for every input, and a threshold turns the score into a decision. Move the threshold one way and more fakes are caught but more real inputs are flagged; move it the other way and the reverse happens. The equal error rate is the error at the single threshold where the two kinds of error coincide. The ASVspoof 2021 evaluation plan defines it in those terms: EER "corresponds to CM threshold τEER at which the miss and false alarm rates equal each other", where a miss is a spoofed input accepted as genuine and a false alarm is a genuine input rejected (ASVspoof 2021 evaluation plan).

Because the two rates are equal at that point, one number describes both: an EER of 15.64 percent means that, at the chosen threshold, 15.64 percent of fakes passed and 15.64 percent of genuine recordings were rejected. The same plan notes that the exact crossing may not exist, because the two rates "change in discrete steps" as the threshold moves across a finite set of scores, so scoring code takes the threshold that brings them closest or interpolates. Lower is better, and 50 percent is chance.

02How does EER differ from AUC and the false positive rate?

AUC summarises every threshold at once: it is the area under the curve traced by the true positive rate against the false positive rate, and it says how well a detector ranks fakes above reals in general. EER picks one point on that same curve, the point where the curve crosses the line of equal errors. A detector can have a high AUC and a poor EER if its curve is shaped so that the equal-error point sits far from the corner. The AUC score explained page and the ROC curve for deepfake detection page cover the whole curve; this page covers one point on it.

The false positive rate is the other half of the EER definition. A deployment rarely wants equal errors: a bank may accept more missed fakes to keep false alarms on customers low, and a court may accept the reverse. EER is useful because it is the one threshold that needs no choice about costs, so two systems can be compared without agreeing on which error matters more. That is its strength and its limit. ASVspoof 5 says so directly: past editions used EER as the primary metric "for the comparison of spoofing countermeasures", and the 2024 edition moved its main deepfake track to a detection cost function with explicit priors and costs while keeping EER as a reported metric.

03Where is EER reported?

Three places in the deepfake literature report it, with different weight.

The voice benchmarks made EER primary where no second system is involved. The ASVspoof 2021 journal paper puts it in one sentence: "the DF task does not involve the use of an ASV system", and where there is no speaker verifier to combine with, "that for the DF task is the CM equal error rate (EER)" (arXiv 2210.02437). The LA and PA tasks keep a speaker verification system in the loop and are scored by the tandem detection cost function instead, with EER as a secondary figure. The voice deepfake benchmarks page describes the three tasks.

Face video benchmarks compute EER but do not lead with it. DeepfakeBench lists "frame-level AUC, video-level AUC, ACC (fake and real), EER, PR, and AP" as its supported metrics, and its main results table reports frame-level AUC for 15 detectors trained on FaceForensics++ c23 (DeepfakeBench README). The best deepfake detection model page reads that table by column; none of its numbers is an EER.

In biometrics the vocabulary is older and larger. ISO/IEC 19795-1:2021, "Biometric performance testing and reporting, Part 1: Principles and framework", "specifies performance metrics for biometric systems" and sets the reporting rules that face recognition evaluations follow; the metrics themselves (false match rate, false non-match rate and their 1:N counterparts) are defined in the standard's text, which is sold rather than published, so this page cites it by title and scope only (ISO/IEC 19795-1:2021).

04What did the ASVspoof 2021 deepfake task find?

The DF task asked systems to detect compressed manipulated speech of the kind posted online, with no speaker verification system involved, and scored them by pooled EER on an evaluation set of 14,869 genuine and 519,059 spoofed trials. The best challenge system, T23, reached 15.64 percent. The next two, T20 and T08, reached 16.05 and 18.30 percent. The two strongest baselines, B04 (a RawNet2 model on raw waveforms) and B03 (LFCC features with an LCNN-LSTM), reached 22.38 and 23.48 percent, and 18 submitted systems beat the best baseline (arXiv 2210.02437, Tables V and XVI).

The gap between the progress set and the evaluation set is the finding most worth quoting. T23 scored 0.24 percent EER on the progress set and 15.64 percent on the evaluation set; the paper reports that 23 of 33 systems were under 10 percent EER on the progress subset, yet every system was above 15 percent on the evaluation subset, whose spoofed speech came in part from source data no challenge had used before. Post-challenge work later reached 2.85 percent EER on the same DF evaluation data (Table VIII), under training conditions the paper lists separately. The paper's abstract draws the lesson: detection solutions "lack generalization across different source datasets".

05How should an EER be read?

Four conditions attach to any EER, and a figure quoted without them cannot be compared. The evaluation set, since the same system scored 0.24 and 15.64 percent on two subsets of one challenge. Whether trials were pooled across conditions or scored per condition, since ASVspoof reports pooled EER and its codec breakdowns separately. Whether a second system was in the loop, since the LA and PA tasks report EER only as a secondary figure beside t-DCF. And the level of the decision, utterance, frame or video, since a frame-level EER and a video-level EER on the same clips are different numbers, as the FaceForensics++ compression levels page shows for accuracy.

The quotable caveat is the one the benchmarks themselves state: an EER measured where the training and test data share a source does not predict the EER on recordings from a new source. Legal uses of a detector score add a further condition that no metric captures; the is a deepfake illegal page describes how the statutes define the content they cover.