Common sense reasoning for deepfake detection: the ECCV 2024 paper explained

This page belongs to the deepfake detection hub and sits next to the papers reading list. It reports what the paper proposes, its dataset and its results as the authors state them, with the conditions of each number.

01What problem does the paper address?

Most detectors are binary classifiers that output one score, and explain themselves, if at all, with a saliency map. The authors argue that such features "may fall short in representing unnatural 'non-physical' semantic facial attributes", and name blurry hairlines, double eyebrows, rigid eye pupils and unnatural skin shading as examples that people notice easily. They propose to model that human judgement by making the detector produce a text explanation, which they call "common sense reasons for labeling an image as real or fake".

The paper appeared in Computer Vision, ECCV 2024 (Lecture Notes in Computer Science, DOI 10.1007/978-3-031-73223-2_22) and was posted to arXiv as 2402.00126, with a first version on 31 January 2024 and a second on 18 July 2024. The figures below come from the second version.

02What is the DD-VQA task, and how was the dataset built?

DD-VQA, short for Deepfake Detection VQA, asks a model to answer a question about a face image. A general question asks whether the whole image looks real or fake. A fine-grained question asks about one component: skin, eyebrows, eyes, nose or mouth. An answer starts with the real or fake judgement and continues with a reason such as overlapping eyebrows or a boundary on the forehead.

The images are single frames from FaceForensics++: the paper counts 5,000 videos in it, 1,000 real and 4,000 fake, made with four manipulation methods. The authors extract one frame from each video, crop the face to 480 by 480 pixels, and collect answers from crowd workers. Three annotators answer each question, and the majority real or fake choice is kept. Annotators pick reasons from a vocabulary in the paper's Table 1, grouped as eyebrows, skin, eyes, nose, mouth with teeth and chin, and others, and can add their own.

The split follows the FaceForensics++ train and test ids.

03How is the model built?

The model is a vision and language transformer. The authors fine-tune BLIP, starting from its base weights with a ViT-B/16 image encoder, for 300 epochs on three GPUs over about 72 hours. Two contrastive losses are added to the usual language modelling loss: one on text, one on images, so that the model separates real and fake features in both modalities. The variants in the paper are BLIP (the baseline), BLIP-T (text loss), BLIP-I (image loss) and BLIP-TI (both).

The second use of the model is as a feature source. The vision representation learned on DD-VQA is added to the representation inside an existing detector, so the detector's own features are combined with features that were trained to support textual reasons. The paper describes this as model agnostic and tests it on four detectors.

04What results does the paper report?

On the DD-VQA task itself, detection is read from the generated text, so the paper gives accuracy, recall, precision and F1 and no AUC. In its Table 2 the BLIP baseline scores an accuracy of 0.8168 and an F1 of 0.8642. BLIP-TI, with both contrastive losses, scores an accuracy of 0.8749 and an F1 of 0.9007, with recall 0.9341 and precision 0.8697. The text calls this "an improvement of nearly 3% in F1 and 6% in accuracy"; the table values differ by 3.65 points in F1 and 5.81 points in accuracy.

For detectors, Table 3 trains each on FaceForensics++ at the low quality c40 setting and tests on that set, on Celeb-DF and on WildDeepfake. The table lists AUC in percent with and without the DD-VQA representation.

The gains on Celeb-DF are 2.50, 2.20, 1.50 and 0.80 points for Xception, HiFi-Net, RECCE and SBIs. The paper also reports, in Table 5, that on general questions Efficient ViT scores an accuracy of 0.6849 and Convolutional Cross ViT 0.7763, against 0.8719 for BLIP-TI, and that training with explanations gives higher detection performance than training on the real or fake answer alone.

05Why do these numbers differ from other detection tables?

Three conditions set them apart from the DeepfakeBench scores on the results page. The detectors here are trained on FaceForensics++ c40, the low quality release (see the compression levels page), where DeepfakeBench trains on c23. The paper reports accuracy, AUC and EER on its own test splits, where the DeepfakeBench headline is frame-level AUC. And the SBIs row follows the self-blended images method: the paper notes that SBIs "only uses real images from FF++(raw) for training", so its starting point is far above the others.

The paper also prints two values for BLIP-TI accuracy: 0.8749 in Table 2 and 0.8719 in Table 5, where the text says the comparison uses general questions. Treat the two as measuring different question sets, and quote the table you took the number from.

06What limits do the authors state, and what is missing?

The paper's Limitations paragraph lists three: "Our method may generate less effective answers when applied to high-quality deepfake images designed to deceive humans"; the dataset needs more exploration of other deepfake images such as computer-generated and diffusion-based ones; and other vision and language backbones are untried. The images are FaceForensics++ frames, made with four older methods, so the dataset says little about current generators.

Tests on other generators, including StyleGAN2 and diffusion-based face editing, appear as qualitative examples in the paper's Figure 6, without a score. The dataset address printed in the abstract returned a not found page on 10 November 2026, so a reader who wants the question and answer pairs should look for a current location in the paper's text or the publisher's page before relying on it.