Self-blended images (SBI): a deepfake detector trained on synthetic swaps

This page belongs to the deepfake detection hub. It follows the Face X-ray blending detector, the 2020 method that first trained on blended real faces, and it feeds the cross-dataset drop explained page, because SBI was built to shrink that drop.

01What is a self-blended image?

A self-blended image starts from a single pristine face and produces a fake with no second person involved. The paper's abstract describes the construction: "SBIs are generated by blending pseudo source and target images from single pristine images", so that the result reproduces "common forgery artifacts (e.g., blending boundaries and statistical inconsistencies between source and target images)" (arXiv 2204.08376). One copy of the face is given small colour, frequency and geometric changes, a mask is drawn from the facial landmarks and deformed, and the altered copy is pasted into the original through that mask with a blending ratio drawn from a short list.

The idea is that a swap made by any method leaves a boundary and a statistical mismatch between the pasted face and its surroundings. A detector that learns those two cues from real faces is not tied to the colour cast of one autoencoder or the texture of one GAN. The authors state the aim as encouraging classifiers "to learn generic and robust representations without overfitting to manipulation-specific artifacts".

02How was the SBI detector trained?

The classifier is an EfficientNet-B4 pre-trained on ImageNet, trained for 100 epochs with the SAM optimiser, a batch of 32 and a learning rate of 0.001. Only eight frames per video are used for training; at inference 32 frames are scored and their predictions averaged into one video score. The training set is the real half of FaceForensics++ and nothing else. The paper's section 4.1 states that "no manipulated images are used even for the model validation in our approach", so no real deepfake touched the model at any stage.

That choice is the method's defining constraint. Every published figure for SBI is therefore a cross-dataset figure in the strict sense: the model never saw the test set's generator, and it never saw any generator at all. The DeepfakeBench v2 README places SBI among its 20 spatial detectors, but the benchmark's published results table still covers the original 15 methods and says it "will update the below table soon", so this page takes SBI's numbers from the paper.

03What did the paper report across datasets?

Table 1 of the paper compiles cross-dataset results for models trained on FaceForensics++ and tested on Celeb-DF, DFD, DFDC, DFDC Preview and FFIW, with the prior methods' numbers cited from their own papers and FTCN re-run by the SBI authors where no number existed. The metric is video-level AUC in percent.

Against the frame-level methods the paper reports an average gain of 4.58 points over the baseline (87.33 against 82.75); against the video-level methods, 6.68 points (86.83 against 80.15). The one set where SBI is second is DFD, where PCL + I2G scored 99.07; the authors note that DFD frames often contain a manipulated face next to pristine ones, which suits a method with a per-face consistency score. The DeepfakeBench rows for the same test sets, measured at frame level and under a different preprocessing, are on this site's results table and are not directly comparable with these video-level values.

04Does the blending idea hold on manipulations other than swaps?

Yes, on the four FaceForensics++ manipulations. Table 2 reports cross-manipulation AUC on the raw release: 99.99 on Deepfakes, 99.88 on Face2Face, 99.91 on FaceSwap, 98.79 on NeuralTextures and 99.64 on the whole set, against 99.11 for PCL + I2G. Face2Face and NeuralTextures are reenactments rather than swaps, and both still leave a blended region behind, which is why the blending cue carries over.

The paper also separates the data from the network. A ResNet-34 trained on SBI averaged 78.54 across Celeb-DF, DFDC and DFDC Preview where the same network trained on I2G averaged 66.61, and an EfficientNet-B1 trained on SBI averaged 88.77 where the B4 averaged 90.95. The training data, not the backbone, carries most of the gain, the same reading the open-source face recognition toolkits page gives for verification models trained on different sets.

05How does SBI read on the DF40 protocols?

DF40 ran SBI without retraining it, because "the original SBI paper only uses BI for training", and scored it with frame-level AUC beside detectors trained on the DF40 fakes (arXiv 2406.13495). The comparison is uneven by design: the other rows were trained on one forgery type of DF40, SBI on none.

On the unknown-domain protocol SBI scored 0.764 on DeepFaceLab swaps and 0.402 on HeyGen talking-head video, and its average of 0.514 sat below Xception's 0.546 and CLIP's 0.692. DF40's authors read this as Finding 6: "SBI could be viewed as an 'anomaly detection' model", one that scores a face by its distance from real faces, so that fakes far from the real distribution are caught and fakes close to it are not. The DF40 protocol tables carry the full rows, and the AI video generators and detection page follows the same generated-video sets into the non-face benchmarks.

06Why numbers differ between sources

The SBI paper reports video-level AUC after averaging 32 frame scores; DF40 and DeepfakeBench report frame-level AUC. The SBI paper tests on FaceForensics++ raw for the cross-manipulation table; DF40 builds its fakes on the FaceForensics++ and Celeb-DF footage with its own 40 methods. And the FTCN values on DFD, DFDC and FFIW in Table 1 come from the SBI authors' runs of the official code, not from the FTCN paper. A reader comparing 93.18 on Celeb-DF with DF40's 0.679 on the Celeb-DF domain is comparing a different metric, a different fake set and a different base of frames.