Deepfake detection with CNNs: Xception, EfficientNet-B4 and MesoNet scored on one table

This page belongs to the deepfake detection hub and compares the five classifiers DeepfakeBench groups as "naive" detectors. The three detector families page places that group next to the spatial and frequency methods built on top of it.

01What is a CNN deepfake detector?

A convolutional neural network (CNN) is an image classifier that learns filters from pixels; a CNN deepfake detector is one trained on cropped faces with two labels, real and manipulated, and nothing else. DeepfakeBench defines the group in one sentence: a naive detector "employs CNNs to directly distinguish deepfake content from authentic data", and names MesoNet and Xception as the usual examples (section 2). The spatial and frequency detectors in the same benchmark reuse these networks as backbones and add a hand-designed input, loss or module.

The five naive detectors in the benchmark span five years of architecture design. MesoNet and MesoInception (WIFS 2018) are small networks written for the task; CNN-Aug (CVPR 2020) is a ResNet-34 trained with heavy augmentation; Xception (ICCV 2019, as the FaceForensics++ baseline) and EfficientNet-B4 (ICML 2019) are general ImageNet classifiers with their final layer replaced.

02What did MesoNet set out to do?

MesoNet (Afchar, Nozick, Yamagishi and Echizen, 2018) was built for video, where compression destroys the pixel-level noise that classic image forensics relies on. Its authors proposed "two networks, both with a low number of layers to focus on the mesoscopic properties of images", the scale between noise and semantics, and reported detection rates of "more than 98% for Deepfake and 95% for Face2Face" on a dataset they assembled from online videos (arXiv 1809.00888). Meso4 uses ordinary convolutions; MesoInception swaps in Inception modules.

Those figures were on the authors' own data. When the FaceForensics++ paper ran MesoNet on its four manipulations, binary accuracy was 95.23 percent on raw video, 83.10 at c23 and 70.47 at c40, against 99.26, 95.73 and 81.00 for XceptionNet on the same splits (arXiv 1901.08971, Table 1). The deeper, pre-trained network held up better under compression, and DeepfakeBench still describes its Xception entry as the XceptionNet model trained on FaceForensics++, in raw, c23 and c40 variants.

03How do the five compare on one table?

DeepfakeBench trains every detector on FaceForensics++ c23 with the same preprocessing, 32 frames per video, and reports frame-level AUC on six within-domain sets (FF++ c23, FF++ c40 and the four manipulations) and eight cross-domain sets. The within-domain and cross-domain averages for the five naive detectors are below; the complete results table holds the full rows.

Two things stand out. The two ImageNet classifiers are far ahead of the three others on seen fakes, by 0.10 to 0.33 AUC on the within-domain average. And the gap between seen and unseen is large for exactly those two: Xception drops 0.17 and EfficientNet-B4 0.17 from within-domain to cross-domain average, while Meso4 rises by 0.05, because Meso4 does badly everywhere and happens to do well on DeeperForensics-1.0 (0.9113) and Celeb-DF v1 (0.7358).

04Are purpose-built detectors much better than a plain CNN?

Not by much, on this benchmark. The DeepfakeBench authors write that "Naive Detectors (e.g., Xception and EfficientB4), which essentially rely on a straightforward CNN classifier, register high AUC values that are comparable to more sophisticated algorithms" (section 4.2). On the cross-domain average, Xception and EfficientNet-B4 at 0.7718 sit 0.0157 below the best detector in the table, SPSL at 0.7875, and above six of the ten purpose-built methods. UCF, the best within-domain method at 0.9527, beats Xception by 0.0077 on that average.

The authors suggest a reason: the published detectors differ from the baselines in pre-training, augmentation and training data as well as in method, and "the performance gap could be a product of these additional factors rather than the intrinsic superiority of the method". DeepfakeBench's own ablation finds pre-training alone worth about 10 percent AUC for Xception on DFDC Preview and for EfficientNet-B4 on the DFD set (section 4.3).

05Why do Xception and EfficientNet-B4 beat ResNet?

CNN-Aug is a ResNet-34 with JPEG and blur augmentation and lands at 0.8419 within-domain, well below the two other ImageNet networks despite a similar parameter count. DeepfakeBench tested why. It identifies "a common module in EfficientNet and Xception that is not present in ResNet, namely the depthwise separable convolution module", inserts that module into ResNet's first layer and reports improvements on many test sets (section 4.3, Table 4). Deeper ResNets help too: ResNet-50 improves on ResNet-34, but ResNet-152 adds little more.

Pre-training is the other variable. The benchmark initialises ResNet, Xception and EfficientNet from ImageNet weights, while Meso4 and MesoInception have no ImageNet weights and train from scratch, which is part of why the two MesoNet rows sit so far below the rest. Every published Xception score with its training set and compression is on the every published Xception score page, and the matching EfficientNet-B4 row is on the EfficientNet-B4 detector page.

06What does the comparison not tell you?

All five scores come from one training set, FaceForensics++ c23, one metric, frame-level AUC, and one protocol, the average of the top three checkpoints. A video-level score, a different training set or a different compression would move every number, and the benchmark's authors state that frame-level evaluation was chosen "to build a fair basis for comparison among detectors", not because it matches deployment. Nor does any row here say how a CNN does on fakes from a generator released after 2020: the newest test set in the table, DeeperForensics-1.0, dates from that year.

The naive detectors are also the easiest to re-run, because their code and weights are public, where five of the purpose-built methods in DeepfakeBench (Face X-ray, UCF, F3Net, SPSL and SRM) are marked "Unpublished code, reproduced by us" in its Table 1. For a reader who wants one baseline number to compare a new method against, Xception at 0.9637 within and 0.7718 across, trained on FaceForensics++ c23, is the one the field uses. A face recognition reader will notice that the same backbones appear in the biometric error rate reporting conventions of the other field, scored by FNMR at a fixed FMR rather than by AUC; the DeeperForensics-1.0 dataset page shows one unseen set where the shallow MesoNet rows score above the deep ones.

Transformer backbones took over the newer entries in the benchmark; the vision transformers for deepfake detection page reports what their papers claim and under which training set.