Frame-level vs video-level AUC: why the same detector gets two numbers

This page is part of the deepfake detection hub and extends the scoring metrics overview, which defines AUC itself. Here the question is not what AUC measures but what counts as one sample when it is computed.

01What is the difference between frame-level and video-level evaluation?

Every face deepfake detector in the DeepfakeBench table is an image classifier: it takes one aligned face crop and returns one score. A video has hundreds of such crops. The evaluation has to decide whether the unit being scored is the crop or the video. At frame level, each sampled frame carries its own label (real or fake, inherited from the video) and its own score, and the ROC curve is drawn over all frames in the test set. At video level, the frame scores for one clip are reduced to a single number, by averaging in the two papers cited here, and the ROC curve is drawn over videos.

The DeepfakeBench paper names the problem directly: in the literature "some are at the frame level, while others are at the video level, leading to unfair comparisons", and so "Our benchmark currently adopts the frame level evaluation to build a fair basis for comparison among detectors" (Yan et al., 2023). Its implementation samples 32 frames per video for training and testing and reports the average of the top three checkpoints' AUCs. Its repository lists both "frame-level AUC" and "video-level AUC" among the metrics the code can compute, so the choice is a protocol decision, not a limitation of the software.

02Why do the two numbers differ for the same model?

Averaging is a noise filter. A frame classifier makes some errors on individual frames of a fake video, from motion blur, a turned head or a frame where the manipulation left little trace. At frame level those frames are counted as misses. At video level they are averaged with the frames the detector did catch, and a clip with most frames scored correctly ends up on the correct side of the threshold. The same filter can hurt: a fake with only a few badly manipulated frames averages toward real. Video-level AUC is therefore usually higher for a competent detector, but not always, and the gap depends on how many frames are sampled and how errors are distributed across them.

Sampling policy is the second source of difference. DeepfakeBench scores 32 frames per video. LipForensics uses the first 110 frames of each test video, in clips of 25 frames, and states that "we compute video-level measures for fair comparison: we average the model predictions (each prediction corresponding to a frame or a video clip) across the entire video", so that "all models' video predictions are based on the same number of frames" (Haliassos et al., 2021). FTCN follows the same convention: "we report video-level AUC for a fair comparison. For the image-based method, we average the model predictions for each frame across the entire video" (Zheng et al., 2021). Two papers that both say "AUC on Celeb-DF" can thus be scoring different frame counts aggregated differently.

03What do the published numbers look like side by side?

The clearest published comparison is for the Xception baseline trained on FaceForensics++ and tested on Celeb-DF v2 and on the DFDC test data, measured by two groups with two protocols.

For Xception the two protocols land within a fraction of a point of each other on these two sets, which is a useful fact in itself: the two groups used different code, different face crops and different frame budgets, and the frame-to-video change did not move the baseline much. The LipForensics and FTCN rows cannot be put into the DeepfakeBench frame-level table at all, because both models consume a sequence of frames and have no per-frame score to rank; FTCN's input clip is 32 frames at 224 by 224 pixels. The cross-dataset scores by model page keeps the two kinds of row apart for that reason.

WildDeepfake reports the same contrast within one paper, in accuracy rather than AUC. Its image-level baselines scored on single faces, with XceptionNet at 69.25 percent and the authors' ADDNet-2D at 76.25 percent, while its sequence-level models scored on 50-frame clips, with I3D at 62.69 percent and ADDNet-3D at 65.50 percent (Zi et al., 2020). The sequence models were lower there, a reminder that "video-level" is not a synonym for "better".

04Which papers use which level?

GenConViT's own evaluation extracted 15 frames from each test video and averaged the predictions (Deressa et al.); DF40 kept DeepfakeBench's frame-level AUC as its primary metric and reported video-level AUC only for I3D (Yan et al., 2024). Audio has the same issue in a different unit: Deepfake-Eval-2024 cut its audio files into four-second segments and reported performance on the segments, so its audio AUCs are segment-level, not file-level.

05How should a reader handle a paper that does not say?

Treat the level as unknown and the number as not comparable until the method section settles it. The checklist on this site for reading a detection paper puts this question next to the training set and the compression level, because the three together define what a score means. A paper that reports "AUC" with no level, no frame count and no aggregation rule has reported a number whose unit is missing.

When two papers do state their levels and they differ, the honest move is to keep the numbers in separate tables rather than rank them together. That is what this site does on its results page, what DeepfakeBench did by choosing one level for all 15 detectors, and what LipForensics and FTCN did in the other direction by converting every baseline to video level before comparing. No single benchmark score predicts how a detector performs on unseen manipulations, and a score whose level is unstated does not even predict how the same detector performs on the same set under the other protocol.

06Where does this matter outside the tables?

In three places on this site. The 25 paper reading list marks which rows are frame-level DeepfakeBench reproductions and which are the papers' own video-level figures. The EU AI Act Article 50 page records that the regulation's provider duty is pitched at marking content rather than at any detection accuracy level, which is consistent with a field whose headline numbers change with the unit of evaluation. And face recognition has the same unit problem in another form: the MegaFace retirement page shows rank-1 identification for the same algorithms falling from above 95 percent with 10 distractors to between 35 and 75 percent with a million, because the gallery, not the model, changed.