Face forgery detection backbones: what detectors reuse from recognition

This page belongs to the deepfake detection hub and is the technical appendix to the bridge page, which follows one pipeline from "who is this?" to "is this real?". Here the question is narrower: which parts of a face recognition system does a forgery detector reuse, and does the choice show in the scores.

01What is a backbone, and what do the two tasks share before it?

A backbone is the convolutional network that turns an aligned face crop into a feature vector; everything task-specific sits on top of it. Before the backbone, face recognition and face forgery detection run the same steps. ArcFace, the reference face recognition method, generates "normalized face crops (112 x 112) by utilizing five facial points predicted by RetinaFace", then feeds them to "the widely used CNN architectures, ResNet50 and ResNet100" and reads out a 512-dimensional embedding (arXiv 1801.07698). DeepfakeBench detects, crops and aligns faces with DLIB, resizes the aligned faces to 256 by 256 for training and testing, samples 32 frames per video, and feeds the crops to a backbone that is "pre-trained" on ImageNet "if feasible" (arXiv 2307.01426).

The difference starts at the output. ArcFace trains its backbone with an additive angular margin loss so that two crops of one person land close together in the embedding space; a detector trains the same kind of backbone with a classification loss so that manipulated crops score high and real ones low. The crop size, the landmarks, the alignment and the pre-training are shared. The weights, after training, are not.

02Which backbones do DeepfakeBench's detectors use?

Table 1 of the DeepfakeBench paper lists each of its fifteen detectors with its backbone. Eight run on Xception: the Xception baseline itself, DSP-FWA, FFD, CORE, UCF, F3Net, SPSL and SRM. MesoNet and MesoInception use the small CNN their authors designed. CNN-Aug uses ResNet, EfficientNet-B4 uses EfficientNet, Capsule uses a capsule structure over a VGG19 network, Face X-ray uses HRNet, and RECCE uses its authors' own network. The paper groups them as naive detectors, which "directly detect forgeries without relying on additional manually designed features", spatial detectors, which build "upon the backbone of CNNs used in the Naive Detector" and add hand-designed algorithms, and frequency detectors, which work on "information from the frequency domain".

The count is partly the benchmark's choice. For DSP-FWA, whose official code lacked a training routine, the authors write that "we use the Xception network [5] as the backbone. This choice is to ensure consistency in the benchmark by using the same backbone network across different detectors". Five detectors (Face X-ray, UCF, F3Net, SPSL, SRM) are marked as reproduced from unpublished code. So the figure above describes the benchmark's implementations, which follow the papers where the papers specify a backbone and standardise where they do not. The DeepfakeBench results and their conditions are read with that in mind: a comparison between two Xception-based detectors is a comparison of methods, not of networks.

03Why Xception?

Xception is an ImageNet classifier, not a face network. Chollet (2017) proposed it as an architecture in which "Inception modules have been replaced with depthwise separable convolutions"; with "the same number of parameters as Inception V3" it "slightly outperforms Inception V3 on the ImageNet dataset" and does better still on a 350 million image, 17,000 class set (arXiv 1610.02357). The FaceForensics++ paper adopted it in 2019 after finding that "a XceptionNet classifier shows the best performance in each test" against five other detectors, reporting 99.26 percent accuracy on raw video and 81.00 percent at low quality when trained on all four manipulations (arXiv 1901.08971), and the field followed.

DeepfakeBench tested whether that habit is justified. Comparing Xception, EfficientNet-B4 and ResNet34 inside the same detectors, the authors found that "Xception and EfficientNet-B4 consistently outperform ResNet34, despite having a similar number of parameters", and they trace part of the gap to "a common module in EfficientNet and Xception that is not present in ResNet, namely the depthwise separable convolution module". Pre-training mattered too: "the pre-trained models can largely outperform their non-pre-trained counterparts, especially in the case of Xception (about 10% in DFDCP)". The pre-training is ImageNet, not face recognition; no detector in the table starts from recognition weights.

04Does the backbone explain the scores?

Partly, and the table says how much. In the paper's Table 3, where every detector is trained on FaceForensics++ c23 and scored by frame-level AUC, the Xception baseline averages 0.9450 within its training set and 0.7718 across the eight other datasets, and EfficientNet-B4 averages 0.9389 and 0.7718, the same cross-dataset figure to four decimals. Meso4, on its small designed CNN, averages 0.6097 and 0.6551. Moving from a small purpose-built network to a modern ImageNet classifier is worth about 0.12 of cross-dataset AUC on its own.

The methods built on top of Xception then spread across a narrower band. Among the eight Xception-based detectors the cross-dataset average runs from FWA's 0.7239 to SPSL's 0.7875, with UCF at 0.7801 and the plain baseline at 0.7718. Face X-ray, on HRNet, averages 0.6985 across datasets while scoring 0.9872 on the Face2Face split, the highest figure in that column. The authors' own reading is that "Naive Detectors (e.g., Xception and EfficientB4), which essentially rely on a straightforward CNN classifier, register high AUC values that are comparable to more sophisticated algorithms". The backbone sets the floor; the method moves the result by a few hundredths.

05What carried over from face recognition, and what did not?

Three things carried over. The front of the pipeline, face detection and landmark alignment, is the same operation in both fields; ArcFace aligns on five RetinaFace landmarks and DeepfakeBench on DLIB landmarks, and either would serve the other task. The idea that the backbone is a commodity, chosen for ImageNet performance and fine-tuned for the task, is shared: ArcFace's contribution is a loss function over ResNet, and most detection papers' contributions are an input transform or a loss over Xception. And the failure mode is shared. Recognition saturated LFW and moved to harder protocols; detection scores above 0.93 within its benchmark for ten of fifteen detectors and no higher than 0.7875 on average outside it.

One thing did not. The WebFace260M work that this domain once hosted trained its recognition models on 42 million cleaned faces and evaluated them under a fixed inference time budget, because at that scale the backbone's size is what the budget buys (arXiv 2103.04098). Detection has no equivalent: its training sets are thousands of videos, its backbones are mid-sized ImageNet classifiers, and the limit on its scores is the mismatch between training fakes and test fakes rather than network capacity. A larger backbone does not close the cross-dataset gap, and the generated faces on the how AI faces are generated page show why: the cue a detector learns belongs to one generator family. What the backbone sees also stops at the pixels; a signed record of the file's history, such as Content Credentials (C2PA), is evidence no backbone can read, and the deepfake statistics with a primary source page sets both kinds of figure side by side.