Log loss: how the Deepfake Detection Challenge scored its entries

This page is the scoring entry for the challenge in the deepfake detection hub. The DFDC dataset page covers how the clips were recorded and manipulated; this page covers the metric, the two test sets and the numbers the paper published for the top five.

01What is log loss?

Log loss, also called cross-entropy loss, scores a probabilistic prediction. For each clip the entrant submits a probability that the clip is fake; the loss for that clip is the negative natural logarithm of the probability assigned to the label that turned out to be true, and the reported figure is the mean over all clips. A confident correct prediction costs almost nothing, a confident wrong prediction costs a great deal, and a prediction of 0.5 costs the natural logarithm of 2, about 0.693, whichever label is true. The challenge's Kaggle evaluation page serves its formula only through scripts, so the description here follows the paper's prose, which states that a constant 0.5 "is roughly the score" of 0.69 (arXiv 2006.07397).

Lower is better, and the scale has no fixed ceiling: a single clip predicted fake with probability 0.999 that is in fact real adds a loss near 6.9 to the sum for that clip. That property is what made log loss suitable for ranking a competition, because it rewards calibrated probabilities rather than hard decisions, and it is also why the paper treats the resulting number as relative rather than absolute.

02Why did the organisers rank by log loss?

The paper's metrics section argues that in deployment, where "the ratio of Deepfaked videos to real videos may be less than one in a million", precision matters more than accuracy, and it proposes a weighted precision that rescales a test set's class balance toward that of organic traffic. It then explains the choice for the contest: "models were ranked by log loss, as the weighted PR metric can be extremely small and somewhat noisy", adding that the organisers "only needed a relative measure of performance to rank competitors, rather than an absolute measure".

So the leaderboard number answers one question, which entry predicted the private set best, and not another, how well a detector would work on a platform. Section 6 of the paper reports weighted precision and recall figures beside the log loss for that second question. The scoring metrics for deepfake detectors page sets log loss next to AUC, EER and false positive rate, and explains why a threshold-free summary and a threshold-bound one can disagree.

03What were the public and private test sets?

The challenge used two hidden sets. The public test set held 4,000 ten-second clips, 2,000 of them deepfakes, from 214 subjects absent from the training data, with one generation method, StyleGAN, that did not appear in training and augmentations applied to roughly 79 percent of clips. It ranked the public leaderboard while the competition ran, and entrants could tune against it.

The private test set held 10,000 ten-second clips, again half deepfakes. Half of the set was "organic content found on the internet"; the other half was unseen footage from the challenge's own source videos, assembled from 260 new subjects and built like the public set except that StyleGAN was absent. Augmentations covered about 79 percent of clips, with two new ones, a dog mask and a flower crown filter, that no entrant had seen. All final evaluations ran on a single V100 GPU with a 90-hour limit for the 10,000 clips; most entries finished in under 10 hours, about 3.6 seconds per video.

04What did the top five score?

Table 2 of the paper lists the five winning entries with their overall log loss, their log loss on the challenge-footage half and their log loss on the real-video half, plus precision at three recall levels on the real videos. The first-placed entry was submitted under an individual's name; the others under team names.

Two readings follow. The five overall scores sit within 0.0092 of one another, so the ranking separated near-equal systems. And every entry scored three to four times worse on the real-video half than on the challenge-footage half: the winner's 0.1983 on footage like its training data became 0.6605 on organic video, close to the 0.69 that a constant guess would earn. The third-placed entry had the best footage score, 0.1703, and the worst real-video score, 0.7039, which is the pattern this site's unseen generators page documents on face images.

05How did the field as a whole do?

The paper reports that 60 percent of all private-set submissions scored a log loss "lower than or equal to 0.69", the constant-guess level, and that "many submissions were simply random". Good public-set scores correlated with good private-set scores, so tuning against the public leaderboard did not mislead the strongest teams, even though the private set added organic video and new distractors. Each team could enter two submissions for final evaluation.

On the real-video half alone the best models reached an average precision of 0.753 and a ROC-AUC of 0.734, which the authors read as evidence that a model trained only on the challenge data "can generalize to real 'in-the-wild' Deepfake videos". Later in-the-wild benchmarks found larger drops: Deepfake-Eval-2024, built from fakes that circulated in 2024, reports that open-source video detectors lost about 50 percent of their AUC against their original benchmarks (Chandra et al., 2025). The two results measure different years and different generators, and both stand with their conditions.

06What does a log loss figure not tell you?

A log loss of 0.4279 says nothing about accuracy at any threshold, nothing about the false positive rate a platform would see, and nothing about performance on fakes made after 2020. It also depends on calibration: two models with the same ranking of clips can score different log losses if one is over-confident. The paper's own caveat applies to every number on this page: the log loss ranked competitors against one another on one test set in 2020, and "Deepfake detection is extremely difficult and still an unsolved problem".

The same lesson appears in audio. The ASVspoof 2021 results ranked speech countermeasures by t-DCF and EER rather than log loss, and its deepfake task showed the same gap between familiar and unfamiliar sources. Official bodies that count fraud publish no detection scores at all, as the measurement of deepfake fraud page sets out, which leaves benchmark figures like these as the only public measure of how well detectors work.