Evaluation protocols: in-dataset, cross-dataset and cross-manipulation

This page is part of the deepfake detection hub. It defines the protocols that the cross-dataset generalization page quantifies, so that a reader of any results table knows which question each column answers before comparing two numbers.

01What is an evaluation protocol?

A protocol is the rule for how the training data and the test data are drawn. It states which dataset each comes from, which manipulation methods appear in each, how the data is compressed, and how the score is aggregated, per frame or per video. Two detectors can only be compared when they were scored under the same protocol, and one detector's score only means something once the protocol is named beside it. DeepfakeBench was built on that point: its authors note "an inconsistency in the usage of these evaluation metrics in the community, some are at the frame level, while others are at the video level, leading to unfair comparisons", and fix frame-level AUC, 32 sampled frames per video, Dlib alignment and a 256 by 256 crop for every detector (arXiv 2307.01426).

The protocol is not the metric. AUC, accuracy and equal error rate are ways to summarise a score distribution; the protocol decides which images the scores were computed on. Choosing both is the first step in evaluating a detector.

02In-dataset: does the detector learn this dataset?

The in-dataset protocol, which DeepfakeBench calls within-domain, trains and tests on splits of one dataset. The authors describe its purpose as assessing "the performance of the model within the same dataset". In DeepfakeBench every detector is trained on FaceForensics++ at compression c23 and tested on six within-domain columns: the whole c23 test split, the c40 split, and the four manipulation methods one at a time. Most detectors do well here: the paper reports that "a majority of the detectors performed commendably, evidenced by high within-domain AUC", with UCF, Xception, EfficientNet-B4 and F3Net averaging 0.9527, 0.9450, 0.9389 and 0.9449 across the six columns in Table 3. The paper's running text gives UCF's average as 95.37 percent while its Table 3 prints 0.9527; this site uses the table throughout, as the results page does.

An in-dataset score answers one question: given this dataset's generators, source footage and compression, can the detector separate real from fake? It does not say whether the detector learned forgery traces or dataset traces. The same paper's Table 3 shows that distinction is real: Xception scores 0.9637 on FF++ c23 and 0.8261 on the c40 split of the very same videos, so a change in compression alone removes 0.14 of AUC.

03Cross-dataset: does it survive a different source?

The cross-dataset protocol, cross-domain in DeepfakeBench, trains on one dataset and tests on another. The paper tests the FF++ c23 models on eight other sets: Celeb-DF v1 and v2, DeeperForensics-1.0, the Google DFD set, DFDC, the DFDC preview, FaceShifter and UADFV. Xception falls to 0.7365 on Celeb-DF v2 and 0.7077 on DFDC; its cross-domain average is 0.7718 against a within-domain average of 0.9450. The best cross-domain average in the table is SPSL at 0.7875, and the paper's own reading is that naive CNN classifiers "register high AUC values that are comparable to more sophisticated algorithms". The full table, column by column, is on the published results page.

Cross-dataset answers a different question: does what the detector learned transfer to new identities, new footage, new generators and new compression at once? Because all of those change together, a cross-dataset drop cannot be attributed to any one of them. That is the gap the third protocol closes.

04Cross-manipulation: which forgery did it learn?

The cross-manipulation protocol holds the dataset fixed and changes only the forgery method. DeepfakeBench trains on one FaceForensics++ method and tests on the other three, so that "only the forgery algorithm is altered. Other factors such as background and identity remain consistent across all the different forgeries". The result is a heat map rather than a table, and the authors' summary is blunt: detectors trained on the Deepfakes method "exhibit excellent performance on the FF-DF test data" but their "performance significantly deteriorates when faced with FF-FS forgeries", and the NeuralTextures test data "poses challenges for almost all detectors".

DF40 (2024) turns this into numbered protocols with published tables. Its Protocol 1, cross-forgery, trains on one of three forgery families within the FaceForensics++ domain, face swapping, face reenactment or entire face synthesis, and tests on the other two. Xception trained on face swapping scores 0.991 on face swapping, 0.892 on reenactment and 0.810 on entire face synthesis; trained on reenactment it scores 0.670 on synthesis (arXiv 2406.13495). Because the identities and footage are shared, these drops measure the forgery family alone. The hub's page on face swaps and reenactment explains what separates the families the detector is asked to bridge.

05DF40's four protocols, and the unknown domain

DF40 names "four standard protocols: Cross-forgery evaluation (Protocol-1), Cross-domain evaluation (Protocol-2), Toward unknown forgery or domain evaluation (Protocol-3), and One-Verse-All (OvA) evaluation (Protocol-4)". Protocol 2 trains on FaceForensics++ and tests on the same forgery families reproduced on Celeb-DF footage: Xception trained on face swapping scores 0.922 on swaps, 0.657 on reenactment and 0.642 on synthesis there, an average of 0.740 against 0.898 in Protocol 1. Protocol 3 tests on generators and footage absent from training entirely: the same Xception model scores 0.882 on DeepFaceLab output and 0.394 on HeyGen output, an average of 0.546 across the unknown set. Protocol 4 trains the Xception baseline on one fake and tests it on the remaining fakes, drawn as a heat map in which "'White' means 0.5 AUC (by chance)".

Sources: DeepfakeBench Table 3 (arXiv 2307.01426); DF40 Tables 3, 4 and 5 (arXiv 2406.13495); read 9 November 2026. Frame-level AUC throughout.

06Why the protocol must travel with the number

A detector described as reaching 0.96 AUC and one described as reaching 0.39 AUC can be the same model with the same weights. Xception trained on FaceForensics++ is both, depending on whether the test set is the FF++ c23 split or HeyGen clips in DF40. Any sentence that quotes a detection score without its protocol is therefore incomplete, and any comparison across papers must first check that the protocols match: same training set, same test set, same compression, same frame or video aggregation. A paper's experimental setup section is where each of those is usually stated.

Two further protocols sit outside this page. Tests on fakes collected from the internet, covered on the in-the-wild page, change everything at once and add unknown platform processing. And presentation attack detection uses its own protocol vocabulary, described on the APCER and BPCER page, where OULU-NPU's four protocols vary session, attack instrument and camera in the same stepwise way DF40 varies forgery and footage. Human judges have been tested under protocols of their own, with chance levels that differ by task design; the human detection studies table records them. The vocabulary for face matching, where the protocol fixes a false match rate rather than a split, is on the 1:1 versus 1:N recognition page.