AI video generators and detection: benchmarks and scores per generator

This page belongs to the deepfake detection hub. It extends the synthetic faces from GAN and diffusion models page from still images to video, and it pairs with the unseen generators page, which covers the same question for face forgeries.

01What counts as an AI-generated video?

Three kinds of video carry the label, and the benchmarks keep them apart. A face swap or reenactment edits a real recording of a person; the swap and reenactment categories page covers those, and most published detectors were built for them. A text-to-video (T2V) model renders a whole clip from a prompt with no source footage; an image-to-video (I2V) model animates a still. GenVideo's authors state the difference as the reason for their set: "Unlike deepfake video datasets ... which focus on human face videos, GenVideo encompasses a broad spectrum of scene contents and motion variations" (arXiv 2405.19707).

The distinction decides which detector applies. A face detector crops a face and scores it; a clip of a street or an animal has no face to crop. The three benchmarks below sit at different points on that line: GenVideo is mostly non-face, DF40 is face only with two generated-video methods, and Deepfake-Eval-2024 is whatever circulated.

02Which generators does GenVideo contain?

GenVideo is "the first AI-generated video detection dataset" at million scale, with 1,078,838 generated and 1,223,511 real videos in all. The training set holds 2,294,594 clips: 1,213,511 real clips from Kinetics-400 (260,232) and Youku-mPLUG (953,279), and 1,048,575 fake clips from ten generators. The test set holds 19,588 clips: 10,000 real clips from MSR-VTT and 8,588 fakes from ten different generators plus a scraped set.

The paper's section 1 gives the training fake count as 1,081,083 where its Table 1 and section 4 give 1,048,575; this page uses the table. The test generators were chosen so that none appears in training, which makes every test score a cross-generator score by construction. The clips are short, one to four seconds for most generators, and the Sora subset is the smallest at 56 clips.

03What did detectors score on GenVideo?

The paper's main protocol is the many-to-many task: train on the ten training generators, test on each held-out generator, report recall, F1 and average precision. Video models beat image models, the authors write, "because video models can model temporal sequences". Average F1 over the ten held-out sets runs from 0.9020 (DeMamba-XCLIP-FT) and 0.8955 (VideoMAE-B) through 0.8755 (FTCN-CLIP-B), 0.8024 (F3Net) and 0.7662 (XCLIP-B-FT) down to 0.7164 for NPR, an image-level method. Adding the DeMamba module to XCLIP raised recall, F1 and average precision to 0.9392, 0.9020 and 0.9710, gains of 11.26, 17.72 and 12.02 points.

The per-generator columns are the useful part. On Sora, F3Net scored an F1 of 0.5000, NPR 0.2786, VideoMAE-B 0.623, XCLIP-B-FT 0.3147 and the best model 0.6467; FTCN-CLIP-B scored 0.7840 on Sora but 0.2922 on Lavie. A detector that averages 0.90 across ten generators can be near chance on one of them, and the one it fails on is not predictable from the average. The paper's second task degrades the test clips by compression and resizing, the damage platforms inflict that the in-the-wild page describes for faces.

04What do the face benchmarks say about generated video?

DF40 includes two application-made video methods among its 40, both test-only: HeyGen, a text-driven talking-head generator with 50 clips on VFHQ footage, and DeepFaceLab, a face swap application, with 100 (arXiv 2406.13495). On the unknown-domain protocol, Xception trained on DF40's FaceForensics++ face swaps scored a frame-level AUC of 0.882 on DeepFaceLab and 0.394 on HeyGen; CLIP scored 0.930 and 0.539; the blending-trained detector on the self-blended image training page scored 0.764 and 0.402. The DF40 unknown-domain protocol page carries the nine-column rows. The text-driven generator was among the three lowest columns for each of the three detectors, and below chance for two of them.

Deepfake-Eval-2024 reports the drop on circulating video without naming generators: GenConViT fell from an AUC of 0.96 on its original benchmark to 0.63, FTCN from 0.87 to 0.50, Styleflow from 0.95 to 0.51 (arXiv 2503.02857). Its error analysis adds the one generator-specific finding: off-the-shelf GenConViT and FTCN "have an average 21.3% lower accuracy on videos which appear to be generated by a diffusion model", identified by watermarks and diffusion-associated visual traits, and the gap narrows to 5.4 percent after fine-tuning on the set. The Deepfake-Eval-2024 page gives the full table.

05Which generators have not been benchmarked?

Any generator released after a benchmark was frozen. GenVideo's newest test generator carries a 2024 date, and its held-out list is fixed; DF40's two video generators are test-only with 50 and 100 clips; Deepfake-Eval-2024 covers what users flagged in 2024 and does not label clips by generator. A detector score on a generator that none of these sets contains is an extrapolation, and the Sora and HeyGen columns show how far an average can be from one held-out generator. The public datasets table on this site records, for each set, which generation methods it holds and in which split.

Two more limits apply to every number above. GenVideo reports F1 and average precision rather than AUC, so its figures are not on the scale of the face tables, and F1 depends on the threshold and the real-to-fake ratio of the test set. And all three benchmarks score a clip on its own; none asks what a false alarm or a miss costs, which is the question the audio field's cost-weighted anti-spoofing metrics were built to answer.