Reproducing a deepfake detection result: a worked example in DeepfakeBench

This page is part of the deepfake detection hub. It is the practical companion to cross-dataset generalisation, which explains why that 0.7365 is so far below the in-dataset score; here the question is what it takes to get the same number again.

01Why do published deepfake scores fail to reproduce?

Most detection papers report a number without the pieces needed to rerun it. The DeepfakeBench README names the problem as "lack of standardization and uniformity in this field", and lists the cure as "a unified data management system that ensures consistent input across all detection models" and "standardized evaluation metrics and protocols". Before it, each group cropped faces its own way, sampled its own frames, chose its own test split and sometimes its own definition of AUC.

The gap is visible in older papers that were careful by the standards of their time. The Celeb-DF authors evaluated nine detectors but wrote that "most of these methods do not have published code for training", so they ran the released weights with default parameters and could not retrain anything (arXiv 1909.12962). The GenConViT authors noted that "the testing sub-datasets of DFDC and FF++ are usually not shared in the scientific literature. Hence, the exact evaluated datasets may differ" (arXiv 2307.07036). Both are statements about missing information, not about the detectors.

02The five things a result has to state

A result is a claim that one model, on one set of inputs, produced one score. To rerun it, a reader needs the training data and its compression (and whether its fakes are manipulated footage or generated media, the distinction on the deepfake versus AI-generated page), the exact list of test items, the steps that turn a video into model input, the formula that turns scores into one number, and either the trained weights or enough training detail to recreate them. Leave any one out and two honest groups will get two numbers, as the Celeb-DF and DeepfakeBench Xception rows show below. The site's how to read a detection paper page uses the same five as its checklist.

03Worked example: Xception from FaceForensics++ c23 to Celeb-DF v2

The target number is the Xception row of DeepfakeBench's Table 3: trained on FaceForensics++ c23, frame-level AUC 0.9637 on the FaceForensics++ test split, 0.7365 on Celeb-DF v2 (arXiv 2307.01426). The repository README reprints the row and states the conditions in one sentence: "The evaluation metric used is the frame-level Area Under the Curve (AUC). In this particular scenario, we train the detectors on the FF++ (c23) dataset" (DeepfakeBench README). The table below traces each of the five through the files that fix it, read on 10 November 2026.

Three further settings in the configuration file matter for a retrain rather than a rerun: Adam at a learning rate of 0.0002 with weight decay 0.0005, batch size 32, 10 epochs, a fixed random seed of 1024, and an ImageNet-pretrained Xception as the starting point. Augmentation is on by default, with horizontal flips, blur and brightness changes each applied with probability 0.5 and JPEG quality drawn between 40 and 100. The README's evaluation command is three lines: the detector config, the list of test datasets and the weights path.

04Which of the five does the reader usually have to supply?

The test split and the preprocessing. Training data are almost always named, the metric is usually named, and weights are released more often than not. What papers omit is the exact list of test videos and the frame sampling. DeepfakeBench supplies both as downloadable artefacts: the README offers the datasets "already preprocessed to cropped faces (32 frames per video) with their masks and landmarks", and the JSON configuration files that fix which video sits in which split. A reader who downloads those and the weights needs nothing else to produce the row. The README's dataset table also fixes the release of each set: c23 only for FaceForensics++, FaceShifter and the Google Deep Fake Detection set, so a DFD score from the benchmark is a c23 score.

The FaceForensics++ authors did the same for their own split. Their dataset README states "We used 720 videos for train and 140 videos for validation as well as testing", and ships the split files (FaceForensics repository). A paper that reports a FaceForensics++ score without saying whether it used that split, or a random one, has left out the item most likely to move the number.

05Why do two sources give different numbers for the same detector?

Two groups have published a frame-level AUC for "Xception trained on FaceForensics++ c23, tested on Celeb-DF", and the numbers are 65.3 percent (Celeb-DF paper, Table 4, 2020) and 0.7365 (DeepfakeBench, Table 3, 2023). Neither is wrong. The Celeb-DF evaluation used the FaceForensics++ authors' released Xception-c23 weights, scored "only on the key frames" of each compressed video, and its own selection of Celeb-DF videos. DeepfakeBench trained a fresh Xception with the settings above, scored 32 sampled frames per video, and used the 518-video test list that Celeb-DF v2 ships. Different weights, different frames, different test items: three of the five differed, and the gap is eight points on a set that was built to be hard, as what makes a deepfake realistic explains.

A third source adds another condition. DF40 reuses the DeepfakeBench code but states that "unlike DeepfakeBench, which does not involve using the validation set, we use the validation set" to pick the checkpoint (arXiv 2406.13495). Checkpoint selection is part of the weights item, and a row from DF40 is not a rerun of a row from DeepfakeBench even when the detector and datasets match. The published cross-dataset AUC rows carry the source and year of every cell for this reason.

06What a result reported this way looks like

A row that can be rerun names the detector and its backbone, the training set with compression, the test set with version and split, the frame sampling, the metric with its level, the source of the weights and the date. In that form the DeepfakeBench row reads: Xception, ImageNet-initialised, trained on FaceForensics++ c23 train split, 32 frames per video at 256 by 256 after DLIB alignment, tested on the Celeb-DF v2 test list in the released JSON, frame-level AUC 0.7365, weights from release v1.0.1, published 2023.

The same template applies to a test on your own data, where the five items become choices rather than lookups; the evaluate a detector page sets them out as a protocol. The README itself warns that its table is "cited from our paper" and that a larger v2 evaluation "will update the below table soon", so a rerun should be matched against the paper's table and dated. No single benchmark score predicts how a detector performs on unseen fakes, and a score that cannot be rerun predicts nothing at all.