EfficientNet-B4 as a deepfake detector: every published score and where it fails

This page belongs to the deepfake detection hub. It follows one row of the benchmark across all ten test sets and sits beside the Xception row, the other ImageNet classifier the field uses as a baseline.

01What is EfficientNet-B4?

EfficientNet (Tan and Le, ICML 2019) is a family of image classifiers built around one idea: scale depth, width and input resolution together with a single compound coefficient instead of enlarging one of them at a time. The authors used neural architecture search to find a small baseline network and scaled it up to B7, which reached 84.3 percent top-1 accuracy on ImageNet "while being 8.4x smaller and 6.1x faster on inference than the best existing ConvNet" (arXiv 1905.11946). B4 is the middle of the family.

Nothing in the design is specific to faces or forgeries. A deepfake detector built on EfficientNet-B4 takes the ImageNet-trained network, replaces the final classification layer with two outputs, real and fake, and fine-tunes on cropped faces. DeepfakeBench lists it as a naive detector for exactly that reason, and notes that "many detectors utilize this architecture as their basic backbone for feature extraction", naming the self-blended images method and the multi-attention detector among them (appendix). Scoring the bare backbone shows how much of a published method's result is the backbone alone.

02How does EfficientNet-B4 score in DeepfakeBench?

Every detector in DeepfakeBench Table 3 is trained on FaceForensics++ c23, 32 frames per video, and evaluated by frame-level AUC averaged over the top three checkpoints. The EfficientNet-B4 row reads as follows; the results by test set page holds the fourteen other rows.

Three cells carry most of the story. Compression alone costs 0.14: the same fakes at c40 score 0.8150 against 0.9567 at c23. NeuralTextures is the hardest of the four seen manipulations at 0.9308, as it is for every detector in the table. And FaceShifter, a face swap that is not among the four training manipulations, is the hardest unseen set at 0.6162, below Celeb-DF v2 and both DFDC sets.

03How close is it to purpose-built detectors?

Closer than the method papers imply. The within-domain average of 0.9389 sits within 0.01 of FFD (0.9434), CORE (0.9431), RECCE (0.9422), SPSL (0.9408), F3Net (0.9449) and SRM (0.9359), and 0.0138 below UCF (0.9527), the best in the table. The DeepfakeBench authors' own summary is that naive detectors "register high AUC values that are comparable to more sophisticated algorithms" and that the remaining gap "could be a product of these additional factors rather than the intrinsic superiority of the method", where the factors are pre-training, augmentation and data (section 4.2).

Across unseen sets the picture is the same. EfficientNet-B4's 0.7718 ties Xception, trails SPSL (0.7875), UCF (0.7801), SRM (0.7760) and FFD (0.7733), and leads CORE, RECCE, F3Net, Capsule, DSP-FWA and Face X-ray. On UADFV it scores 0.9472, the second-best cell in that column after UCF's 0.9528. The CNN detectors compared page sets B4 beside Xception, ResNet-34 and the two MesoNet variants.

04Why do so many detectors build on EfficientNet-B4?

DeepfakeBench's architecture analysis gives two measured reasons. First, "Xception and EfficientNet-B4 consistently outperform ResNet34, despite having a similar number of parameters", and the authors trace part of the difference to the depthwise separable convolution module both networks share and ResNet lacks; inserting it into ResNet's first layer improved results on many test sets (section 4.3). Second, ImageNet pre-training matters more for these two networks than for ResNet: the benchmark reports a gain of about 10 percent AUC for EfficientNet-B4 on the DFD set and for Xception on DFDC Preview when pre-trained weights are used.

A third reason is practical and documented in the benchmark's own text rather than measured: the architecture is public, pre-trained weights are public, and because the benchmark implements it directly, "we can compare the performance of different basic architectures and find the improvement bring by only the architecture". A method that reports a score on an EfficientNet-B4 backbone can be checked against the bare backbone's row above.

05Where does EfficientNet-B4 fail?

On every unseen face swap released after its training set. The row drops from 0.9567 on seen fakes to 0.6955 on DFDC and 0.6162 on FaceShifter, a fall of 0.26 and 0.34 AUC, and the why scores fall on unseen sets page shows that every detector in the table behaves the same way. The cross-domain average of 0.7718 means that, averaged over eight datasets, a random real frame outranks a random fake frame about 23 times in 100.

The row also says nothing about video-level scoring, about training sets other than FaceForensics++ c23, or about fakes from after 2020. The DeeperForensics-1.0 dataset page shows that on one unseen set, where B4 scores 0.8330, the shallow MesoNet classifiers score above 0.91, because what counts as an easy set depends on the match between detector and generator rather than on the detector's depth.

06Why numbers differ between sources

DeepfakeBench's section 4.2 text gives EfficientNet-B4's within-domain average as 93.89 percent, which matches the table's 0.9389; the same sentence gives UCF as 95.37 percent where Table 3 prints 0.9527, and this site uses the table. Scores for EfficientNet-B4 quoted from method papers are usually video-level, on different training sets or with the authors' own augmentation, and are not comparable with the frame-level row here. In face recognition the same network family appears among the WebFace260M baselines (arXiv 2103.04098) scored by false non-match rate at a fixed false match rate, the convention the biometric face recognition standards page describes, so an EfficientNet number from that field is a different measurement again.