Vision transformers for deepfake detection: what the papers report

This page belongs to the deepfake detection hub and reads the transformer rows the way the page on reading a detection paper prescribes: training set, test set, metric, protocol, before the number.

01Why are transformers used for deepfake detection?

A vision transformer splits an image into fixed-size patches, embeds each one, and lets every patch attend to every other. For a forgery detector that structure has a specific appeal: a swapped or reenacted face differs from its background, and the attention map between patches is a built-in measure of which regions agree with which. The UIA-ViT authors state the point as their design reason: the vision transformer "naturally provides the consistency representation from the attention map among patch embeddings", so inconsistency between a manipulated region and the untouched background can be learned without pixel-level forgery masks (arXiv 2210.12752).

The second appeal is pretraining. Transformer backbones usually arrive with weights learned on ImageNet-21K or, for CLIP, on internet-scale image and text pairs, and several papers report that the pretrained feature space generalises to unseen generators better than a network trained from scratch to separate real from fake. That claim is examined below with its numbers.

02Which transformer detectors are in DeepfakeBench?

The DeepfakeBench README (version 2) lists 36 detectors, "28 image detectors + 8 video detectors". Among the 20 spatial detectors it names UIA-ViT, CLIP and Effort; among the 8 video detectors it names TALL, X-CLIP, TimeTransformer and VideoMAE (DeepfakeBench README). Four of the seven are transformer architectures of the video kind, and the README's own update table dates them: TALL to ICCV 2023, Effort to ICML 2025.

The README's results table, however, prints frame-level AUC for the original fifteen detectors only, trained on FaceForensics++ c23: Xception at 0.9637 on FaceForensics++ c23 and 0.7365 on Celeb-DF v2, UCF at 0.9705 and 0.7527, SPSL at 0.8150 on Celeb-DF v1 and 0.7650 on v2. No transformer row appears in that table, so a transformer score quoted "from DeepfakeBench" today comes from the detector's own paper or from a user's run, not from the benchmark's published grid. The DeepfakeBench codebase page explains what the benchmark standardises and why that matters for a comparison.

03What does TALL-Swin report?

TALL (Thumbnail Layout) takes consecutive frames, masks a fixed position in each, shrinks them and tiles them into one 224 by 224 thumbnail, so an image model sees temporal structure without a video architecture. The authors attach it to a Swin-B backbone pretrained on ImageNet-21K, sample 8 clips of 4 frames per video, and compute video-level predictions by averaging over the whole video (arXiv 2307.07494).

Trained on FaceForensics++ HQ, the paper's Table 1 reports video-level AUC on Celeb-DF and DFDC of 82.33 and 72.64 for a plain ViT-B, 83.13 and 73.01 for a plain Swin-B, 86.96 and 74.61 for ViViT, and 90.79 and 76.78 for TALL-Swin, at 47.5 G FLOPs against ViViT's 628 G. Table 3 extends the unseen sets to FaceShifter (99.67) and DeeperForensics (99.62), an average of 91.71 across the four. On the training distribution the picture flips with compression: Table 2 gives TALL-Swin 99.87 AUC on FaceForensics++ HQ and 94.57 on LQ, and the authors write that TALL-Swin "gets unsatisfactory results with the LQ setting". A reader quoting the 90.79 figure should therefore carry three conditions with it: trained on HQ, video-level, Celeb-DF.

04What does UIA-ViT report?

UIA-ViT is a ViT-Base image detector (16 by 16 patches, 12 encoder layers, 224 by 224 crops) trained on all four FaceForensics++ manipulations at HQ. Its cross-dataset table reports frame-level AUC on unseen sets and video-level AUC on the training set:

Source: UIA-ViT, cross-dataset results table (arXiv 2210.12752). The ablation in the same paper is the more informative row for this page: a plain ViT-Base with the same training recipe scores 75.32 on Celeb-DF v1 and 76.25 on v2, so the architecture alone gives roughly Xception-level generalisation and the consistency modules add the rest. Note that the Celeb-DF v2 figures here (82.41) and in the TALL paper (90.79) are not comparable: one is frame level, the other video level, on different crops.

05What do CLIP-based detectors report?

The CLIP route freezes a vision-language transformer and trains almost nothing on top. Ojha, Li and Lee (CVPR 2023) take the final layer of CLIP's ViT-L/14 image encoder, train on ProGAN real and fake images only, and classify with nearest neighbour or a linear probe. Their Table 2 reports a total mean average precision of 93.38 for the linear classifier and 90.32 for nearest neighbour across 19 generators, against 83.58 for the best fully trained convolutional baseline, and 73.44 for a CLIP ViT that was fine-tuned end to end on the same data (arXiv 2302.10174). The paper's reading is that a classifier trained to find fake patterns treats the real class as "a sink class holding anything that is not fake", while a frozen feature space does not. Those are whole-image fakes from GANs and diffusion models, not face swaps, so the result transfers to face deepfakes only as a method.

DF40 supplies the face-video version. Its Protocol 3 table, which trains on FaceForensics++ face swaps and tests on unknown domains, gives CLIP an average frame-level AUC of 0.692 against Xception's 0.546, with 0.930 against 0.882 on DeepFaceLab fakes and 0.539 against 0.394 on one text-driven generator; its Protocol 2 average on the Celeb-DF domain is 0.814 against 0.740 (arXiv 2406.13495). The authors list it as their third finding: "CLIP excels in deepfake detection than other baselines (such as Xception)". Effort, the ICML 2025 detector that now heads the DeepfakeBench README, builds on the same idea by freezing the principal components of a pretrained foundation model's feature space and adapting only the rest (arXiv 2411.15633); its abstract gives no score, and none is quoted here. The DF40 page carries the full protocol tables.

06How should a transformer detector's score be read?

Four conditions change a transformer's number more than the architecture does. The training set: every figure above is trained on FaceForensics++, and the DF40 tables show the same backbone moving by 0.2 AUC or more when the training forgery type changes. The compression level: TALL-Swin loses 5.3 AUC points between HQ and LQ on its own training set. The level: video-level AUC averages predictions over a clip and is routinely several points above frame-level AUC for the same model, so UIA-ViT's 82.41 and TALL-Swin's 90.79 on the same dataset are measurements of different things. And the pretraining: a CLIP or ImageNet-21K backbone has seen data the convolutional baseline has not, which is the point of using it but also means "trained only on FaceForensics++" is true of the classifier head, not of the features.

Those conditions are the columns of the cross-dataset rows on this site, and a transformer row is added there only when a paper states all four. No single benchmark score predicts how a detector performs on unseen manipulations, and the transformer papers above show the gap in their own tables: TALL-Swin's 99.67 on FaceShifter and 76.78 on DFDC come from the same trained model.