UADFV and DeepFake-TIMIT: the first two deepfake test sets, by the numbers

This page is part of the deepfake detection hub. The two sets open the deepfake dataset timeline, and the public deepfake datasets compared table places them beside the sets that replaced them.

01Where did UADFV come from?

UADFV was not built as a benchmark. It is the evaluation set of two 2018 papers from the University at Albany on detecting the first generation of face-swap videos. The first, "In Ictu Oculi" (June 2018), looked for a missing physiological signal, "eye blinking in the videos, which is a physiological signal that is not well presented in the synthesized fake videos", using a CNN with an LSTM on the eye region; its authors downloaded 50 videos of one person each, about 30 seconds long, used 40 for training and 10 for testing, and made fakes "using DeepFake with post-processing" (Li, Chang and Lyu, 2018).

The second paper, on inconsistent head poses (November 2018), is the one later dataset papers cite for UADFV and the one that states its size: "this dataset contains 49 real videos, which were used to create 49 Deep Fake videos", with "an average length of approximately 11.14 seconds" and "a typical resolution of 294 × 500 pixels" (Yang, Li and Lyu, 2018). The cue is that a swapped face is spliced into the original frame, so the 3D head pose estimated from the central facial landmarks disagrees with the pose estimated from the whole face. An SVM on that difference reached a frame-level AUROC of 0.89 on UADFV and 0.974 when frame scores were averaged per video, trained on frames from 35 real and 35 fake videos (21,694 images) and tested on the remaining 14 and 14 (11,058 frames). The Celeb-DF paper records the fakes as made with FakeAPP and the release month as November 2018 (Li et al., 2020, Table 1).

02Where did DeepFake-TIMIT come from?

DeepFake-TIMIT came from a different question: whether face recognition systems could be fooled by swapped faces. Korshunov and Marcel at the Idiap Research Institute took VidTIMIT, a lab-recorded set with "10 videos for each of 43 subjects" reciting short phrases to camera, and "manually selected 16 pairs in such a way that subjects in the same pair have similar prominent visual features, e.g., mustaches or hairs styles" (Korshunov and Marcel, 2018). Each pair was swapped in both directions with an open-source GAN-based face swap, giving fakes for 32 of the 43 subjects.

Two models were trained per pair. The low quality model used 64 by 64 faces, about 200 frames at 4 frames per second and 100,000 iterations, about 4 hours on one GPU; the high quality model used 128 by 128 faces, about 400 frames at 8 frames per second and 200,000 iterations, about 12 hours. The result was "320 videos corresponding to each version", 640 fakes in all, with the original audio track left untouched. The recognition result was the paper's headline: using the equal error rate threshold from the licit scenario, the VGG-based system accepted 88.75 percent of low quality swaps and 85.62 percent of high quality swaps, and the Facenet-based system 94.38 and 95.00 percent, against licit equal error rates of 0.03 and 0.00 percent.

03The two sets by the numbers

The last row is from the DeeperForensics-1.0 user study, which showed 30 clips from each dataset to 100 participants, most of them computer vision researchers, and counted a score of 4 or 5 on a five-point scale as "real"; FaceForensics++ scored 8.4 percent on the same test and DeeperForensics-1.0 itself 64.1 percent (Jiang et al., 2020, Table 3). The Celeb-DF paper groups UADFV, DeepFake-TIMIT and the FaceForensics++ DeepFakes subset as "the first generation of DeepFake datasets" and motivates its own set by their "low visual quality". The counts, generators and rights positions of all of them are on the Celeb-DF page and the FaceForensics++ page.

04Why do subject counts differ between sources?

DeepFake-TIMIT is listed with 43 subjects in the ForgeryNet comparison table and with 32 in the FakeAVCeleb and WildDeepfake tables. Both are right about different things: VidTIMIT has 43 subjects and all of their real videos are in the set, while only the 32 subjects in the 16 selected pairs have swapped versions. The paper's own total also moves: its abstract and method section say 320 fakes per quality level, while its conclusion says "620 Deepfake videos for 16 pairs of subjects", a figure that does not match 2 times 320. This page uses 43 subjects and 640 fakes, the ForgeryNet count, and names the source wherever a figure appears.

05What were the first detectors tested on them?

DeepFake-TIMIT shipped with its own baselines, split by quality. An audio-visual lip-sync detector failed outright, with an equal error rate of 41.8 percent on the low quality fakes, because the swaps kept the original mouth movement and audio. Raw pixel features with PCA and LDA gave 39.48 percent; image quality measures with PCA and LDA 20.52 percent; and image quality measures with an SVM, a method borrowed from presentation attack detection, 3.33 percent on low quality fakes and 8.97 percent on high quality ones. The authors' conclusion was that "GAN-generated Deepfake videos are challenging for both face recognition systems and existing detection methods".

UADFV's first detectors were the two Albany methods above, the blink detector and the head pose SVM at 0.89 frame-level AUROC. Neither approach survived the next generation of generators: the DeepfakeBench results that follow use CNN classifiers, not physiological cues.

06How are they used now?

Both sets are too small to train on and too easy to serve as the headline test, so they survive as unseen test sets. In DeepfakeBench's main table, every detector is trained on FaceForensics++ c23 and tested on eight other sets, UADFV among them: UCF reaches 0.9528 frame-level AUC there, Xception 0.9379, the blending-boundary Face X-ray detector 0.8989 and Meso4 0.7150, and UADFV is the best unseen set for 11 of the 15 detectors (Yan et al., 2023, Table 3). A high cross-dataset score on UADFV therefore says that a 2018 generator is easy for a detector trained on 2019 fakes, not that the detector generalises; the frame-level AUC table by test set shows the same detectors falling to between 0.5551 and 0.6553 on FaceShifter.

DeepFake-TIMIT appears as a reference point in later papers rather than as a test column. The WildDeepfake paper reports that image-level baselines scoring 90 to 99 percent accuracy on the FaceForensics++ DeepFakes subset or on DeepFake-TIMIT fall below 70 percent on fakes collected from the internet (Zi et al., 2020). And UADFV's real footage has a second life: DF40 builds its 100-video DeepFaceLab test set on UADFV source videos, using the 2018 real clips as an unknown domain for 2024 detectors (Yan et al., 2024, Table 2).