How deepfake detection works: frames to score, and the cues models learn

This page sits in the deepfake detection hub as the explainer behind the scores. Where a detector's cue comes from, and where it goes missing, is the subject of why detectors fail in the wild.

01The pipeline from clip to verdict

Detection is framed as a per-frame binary classification problem. The FaceForensics++ paper (ICCV 2019) set that convention: "We cast the forgery detection as a per-frame binary classification problem of the manipulated videos", with each frame cropped to the tracked face (arXiv 1901.08971). A clip is therefore a bag of face crops, each scored on its own, and the video result is whatever rule combines them.

DeepfakeBench makes the steps explicit so that different models see identical input. Its README describes "a sequential workflow for face detection, alignment, and cropping", with faces cropped at 32 frames per video together with masks and landmarks, and the paper adds that detection, cropping and alignment use Dlib and that aligned faces are resized to 256 by 256 pixels for training and testing (arXiv 2307.01426). The model box is the only one a method author controls. The DeepfakeBench page covers the rest of the framework.

02What cues do detectors learn?

The DeepfakeBench paper defines three detector types by their input and design. A naive detector "leverages various CNN architectures to directly detect forgeries without relying on additional manually designed features". A spatial detector "builds upon the backbone of CNNs used in the Naive Detector and further explores manual-designed algorithms to detect deepfake", which in the paper's survey means forgery region location, capsule networks, disentanglement, image reconstruction and blending-boundary methods. A frequency detector "focuses on utilizing information from the frequency domain and extracting frequency artifacts for detection"; SPSL and SRM use phase spectrum analysis and high-frequency noise respectively. The v2 README adds a fourth group, video detectors, which read several frames at once and learn temporal cues.

Counts and names are from the DeepfakeBench README (36 detectors, 28 image and 8 video), read 9 November 2026. The original paper implemented 15 detectors in the first three families; the video family was added in v2.

03How does a per-frame score become a verdict?

The model emits a probability for each face crop. A frame-level metric scores those probabilities directly against the frame labels; a video-level metric first combines them, for example by averaging, into one number per clip. The two are not interchangeable, and the DeepfakeBench paper names the mismatch as a reason published results had been hard to compare: "some are at the frame level, while others are at the video level, leading to unfair comparisons". The benchmark "currently adopts the frame level evaluation to build a fair basis for comparison among detectors", and v2 reports both frame-level and video-level AUC.

The threshold comes last. AUC measures ranking over every possible threshold, so a detector with 0.96 AUC still needs a chosen operating point before it can say "fake" about one clip, and that choice trades false alarms against misses. DeepfakeBench reports AUC, accuracy, average precision and equal error rate, each defined on the metrics page listed below.

04Why does the face crop matter so much?

The crop is not a detail. FaceForensics++ Table 1 reports the same XceptionNet trained two ways: on a crop "enlarged by a factor of 1.3" around the tracked face, and on the full frame. On raw video the face crop reaches an accuracy of 99.26 percent and the full image 82.01 percent; at the heaviest compression the figures are 81.00 and 70.52 percent. The manipulation lives in a few hundred pixels, and a model fed the whole frame spends capacity on background that carries no signal.

Alignment serves the same purpose. Rotating and scaling every face so that the eyes and mouth land in fixed positions removes pose variation the model would otherwise have to learn, and it makes a blending boundary or a mouth-region artefact appear in roughly the same place in every crop. Every published Xception score, with its crop, compression and test set, is collected on the detector page listed below.

05What do the published numbers say about each family?

Trained on FaceForensics++ c23 and scored by frame-level AUC, the families are close in-dataset and spread apart on unseen data. Xception, a naive detector, scores 0.9637 on FaceForensics++ c23 and 0.7365 on Celeb-DF v2. UCF, a spatial detector, has the best within-domain average at 0.9527. SPSL, a frequency detector, has the best cross-domain average at 0.7875 (DeepfakeBench Table 3). The paper's own reading is that naive detectors "register high AUC values that are comparable to more sophisticated algorithms", and that the gap between them "could be a product of these additional factors rather than the intrinsic superiority of the method", the factors being pre-training and augmentation.

All rows are frame-level AUC from DeepfakeBench Table 3 (arXiv 2307.01426, 2023), detectors trained on FaceForensics++ c23. The full DeepfakeBench table with every test set carries the other columns and their dates.

06What this pipeline does not cover

Two neighbouring tasks use similar words and different pipelines. Presentation attack detection decides whether a live person is in front of a camera, is tested under ISO/IEC 30107-3, and reports APCER and BPCER rather than AUC; the PAD standard explained page covers it. Human perception of generated faces is measured by forced-choice experiments rather than by a model; the Which Face Is Real game and its studies page covers that.

The quotable caveat is the one every table on this site carries: no single benchmark score predicts how a detector performs on manipulations it was not trained on. The pipeline above is fixed; the cue the model learns is specific to the fakes it saw.

Detectors that read upsampling artefacts in the frequency domain are explained on the frequency-domain detectors explained page.