How to read a deepfake detection paper: train set, test set, metric, protocol

This page is part of the deepfake detection hub and is the reading guide behind the site's results page, where every row carries those four fields in its own columns.

01The four things to find first

Papers rarely put the four fields in one place. The training set is usually in an "experimental setup" or "implementation details" paragraph, often as a single word such as "FF++" that stands for a specific release at a specific compression. The test sets are the column headings of the main table. The metric is in the table caption, sometimes only there, and the level, frame or video, may be stated once in the setup and never again. The protocol is scattered: compression in the setup, the split in a footnote, and whether a test set was unseen implied by the table's structure rather than said.

Make a list of the four before reading the results. If one cannot be found, the result cannot go into a comparison, however good it looks.

02Train set and test set

The training set determines what the detector learned. For most papers since 2019 it is FaceForensics++, and the FaceForensics++ page explains why "FF++" needs qualifying: four methods or five, c23 or c40, all methods or one. The test sets determine what the score describes. A test set that shares its generators and source footage with the training set measures memorisation of that set's traces; a test set the detector never saw measures generalisation. Papers often report both in one table, in-dataset columns first, and the drop between them is the most informative thing in the paper. Check each test set against the datasets table: its size, its methods and, for the smaller sets, how few videos a high score rests on. UADFV has 49 fake videos.

03Metric and operating point

AUC, accuracy, equal error rate and log loss answer different questions and cannot be compared across each other. AUC averages over every threshold; accuracy depends on one threshold and on how many fakes the test split contains; EER is the error at the threshold where misses equal false alarms; log loss penalises confident wrong answers. Within AUC, frame-level scores each frame and video-level aggregates frames into one score per clip, usually giving a higher number. A paper that says "AUC" without the level, or "accuracy" without the class balance, has left a field blank. The metrics page defines each and shows where each misleads.

04Protocol, compression and splits

Compression is the quiet variable. The FaceForensics++ paper itself shows XceptionNet at 99.26% binary accuracy on raw video, 95.73% on its high-quality release and 81.00% on its low-quality release, on the same fakes (arXiv 1901.08971); a paper trained at c23 and tested at c23 has made a choice that a paper tested at c40 has not. Splits matter because a dataset's official split may be ignored: a detector that trained on frames from a video that also appears in the test split has seen the test set. DeepfakeBench fixes the split per dataset in a configuration file precisely so that this cannot happen silently (arXiv 2307.01426); a paper that does not use a shared harness should state how it split.

05An annotated example table

Rather than invent numbers, take one real row and read it the way the four fields require. Table 3 of the DeepfakeBench paper lists Xception, trained on FaceForensics++ c23, scored by frame-level AUC on fourteen test columns. Train set: FF++ c23, stated once in the caption, "All detectors are trained on FF-c23". Test sets: the column headings, six within-domain (FF++ c23, c40, and its four method subsets) and eight cross-domain. Metric: AUC, with the level given in the text, "Our benchmark currently adopts the frame level evaluation". Protocol: in-dataset for the first six columns, cross-dataset for the rest, c23 training compression throughout.

Read with the fields in place, the row says: 0.9637 on the split it was trained on; 0.8261 when the same methods are compressed harder; between 0.6249 (FaceShifter) and 0.9379 (UADFV) on sets it never saw. Read without them, it says "Xception gets 0.96", which is the ceiling and nothing else. The figure above draws the row.

06From paper to our results table

A row enters this site's results page only when all four fields can be filled from the paper, in the paper's own words, and the value has been matched against the paper on a dated check. That is why most of the table's rows come from one shared evaluation: DeepfakeBench fixed the preprocessing, splits and metric across detectors, so its rows are comparable with each other in a way that numbers from separate papers are not. Where two papers report different values for what looks like the same setting, both rows go in with their sources, and the difference is usually one of the four fields.

07Reproducing a result

The final test of a paper is whether its number can be reproduced under its stated conditions. If the paper released code and weights, the DeepfakeBench test script will score them on any configured dataset; if it did not, the four fields are what another group would need to attempt it. The evaluate a detector protocol on this site turns the same four fields into a procedure for testing a detector on your own data so that the result can be put beside published rows.