DeepfakeBench: a shared codebase for comparing detectors, and how to read its tables

This page is part of the deepfake detection hub. Most of the figures on this site's results page come from DeepfakeBench, so this page explains what the benchmark fixes, what its tables mean, and what they do not.

01What DeepfakeBench is

The paper's abstract states the problem it was built for: "a critical yet frequently overlooked challenge in the field of deepfake detection is the lack of a standardized, unified, comprehensive benchmark. This issue leads to unfair performance comparisons and potentially misleading results" (arXiv 2307.01426). Papers had been reporting numbers produced with different face crops, different frame sampling, different augmentation and different metric definitions, so a method that looked better on paper might have had a better pipeline rather than a better idea.

The benchmark's three stated contributions are "a unified data management system to ensure consistent input across all detectors", "an integrated framework for state-of-the-art methods implementation", and "standardized evaluation metrics and protocols to promote transparency and reproducibility". It is maintained by the Secure Computing Lab of Big Data at the Chinese University of Hong Kong, Shenzhen, released under a CC BY-NC 4.0 licence, with pre-trained weights for each implemented detector (DeepfakeBench repository).

02What it standardises

Preprocessing

Every dataset goes through the same face detection, alignment and cropping, with 32 frames sampled per video in the preprocessed data the authors distribute, plus landmarks and masks. The repository supplies the processed data for FaceForensics++ (c23 only, with its FaceShifter and Deep Fake Detection subsets), Celeb-DF v1 and v2, UADFV, DFDC Preview and the DFDC test data, and notes that copyright stays with each dataset's providers.

Datasets and splits

The original release covered nine datasets: FaceForensics++, FaceShifter, DeepfakeDetection, DFDC Preview, DFDC, Celeb-DF v1, Celeb-DF v2, DeeperForensics-1.0 and UADFV. A JSON configuration per dataset fixes the train, validation and test splits so that everyone tests on the same videos. The datasets page describes each set.

Detectors

The 36 detectors in the current version are grouped as naive (Xception, MesoNet, MesoInception, CNN-Aug, EfficientNet-B4), spatial (20, including Face X-ray, UCF, SBI, SLADD, LSDA and Effort), frequency (F3Net, SPSL, SRM) and video (8, including FTCN, TALL, I3D and VideoMAE). Each is a module in the same training and testing code, so the comparison is between ideas rather than between implementations.

Metrics

The current version reports frame-level AUC, video-level AUC, accuracy for fake and real, EER, precision-recall and average precision. The metrics page defines each; the point of listing them together is that a reader can see the same detector under several definitions instead of the one its authors chose to publish.

03The train-on-one, test-on-many grid

The headline table trains each detector on FaceForensics++ c23 and reports frame-level AUC in two groups: within-domain (FaceForensics++ c23 and c40 and its four method subsets) and cross-domain (Celeb-DF v1 and v2, DeeperForensics-1.0, DeepfakeDetection, DFDC, DFDC Preview, FaceShifter, UADFV), with an average and a count of top-three finishes for each group.

Three rows show the grid's logic. Xception: within-domain average 0.9450, cross-domain average 0.7718. UCF: within-domain average 0.9527, the best in the table, cross-domain 0.7801. SPSL: within-domain 0.9408, cross-domain 0.7875, the best cross-domain average. The within-domain column is nearly flat across serious methods; the cross-domain column is where methods differ, and it is where the field's real progress or lack of it shows. The cross-dataset generalisation page works through the gap; the results page carries the full rows with their dates.

04Reading its tables

Four habits make DeepfakeBench tables safe to quote.

First, note the training set and compression in the caption: these are FaceForensics++ c23 models, and a detector trained on something else would give different numbers everywhere. Second, note the metric: frame-level AUC scores each frame; video-level AUC aggregates per video and is usually higher, and the two are not interchangeable. Third, note the version: the repository says the results shown "are cited from our paper" and that an expanded v2 evaluation "will update the below table soon", so figures quoted from the README are the 2023 numbers until that happens. Fourth, note that bold in the table marks the best method per column, not a recommendation.

The how to read a detection paper page applies these habits to papers that do not use a shared codebase, where the caption often omits one of the four.

05Using it to evaluate your own detector

Because the pipeline, splits and metrics are fixed, a new detector implemented as a DeepfakeBench module can be scored against the published rows without re-implementing the baselines. The repository's test script evaluates a detector's weights on any of the configured datasets, and the release includes pre-trained weights for the implemented methods. The evaluate a detector protocol on this site assumes exactly this kind of fixed harness; DF40, the same group's 40-method dataset, is built to slot into the same workflow for testing against recent generators.

06Results drawn from it on this site

The results page reproduces rows from the DeepfakeBench table with the training set, test set, compression, metric and the date the table was read, and marks them as the paper's 2023 numbers. When the v2 table is published, rows will be re-checked and re-dated, and this page will say so.