Can humans detect deepfake videos? Crowds versus machines, measured

This page sits in the human perception strand of the deepfake detection hub. The human accuracy on synthetic still faces page covers photographs; this one covers moving video, where the only large controlled comparison of people and a model is the study above.

01What did the study measure?

The Deepfake Detection Challenge (DFDC) was a Kaggle competition run on a dataset of more than 100,000 clips recorded with 3,426 paid actors. Groh and colleagues report that 2,116 teams entered and that the winning model scored 65 percent accuracy on the 4,000 holdout videos, half of them fake (PNAS 2022). That model is the "leading model" throughout the paper. How the competition itself scored entries, by log loss on a hidden set, is on the benchmark protocols compared side by side page.

The paper runs two online experiments. Experiment 1 was a two-alternative forced choice: 5,524 people who found the site on their own saw 56 pairs of videos, one real and one deepfake of the same actor, over 26,820 trials. The pairs came from the DFDC training set because the holdout videos were not yet public. Experiment 2 showed a single video and asked for a confidence that it was a deepfake: 9,492 people took part, 304 recruited through Prolific for 6,390 trials and 9,188 organic visitors for 67,647 trials. Its stimuli were 50 videos sampled at random from the holdout set, half fake, plus four clips of two heads of state and an attention check.

02How did individuals compare with the model?

In experiment 1, among the 882 participants who saw at least ten pairs, 82 percent scored above the model's 65 percent. Half the pairs (28 of 56) were identified correctly by more than 83 percent of participants; 12 pairs were identified by fewer than 65 percent. The comparison is indirect, since the model's 65 percent was measured on different videos, which is why the paper's main comparison is experiment 2.

In experiment 2 the model and the participants saw the same 50 videos. Recruited participants identified deepfakes correctly in 66 percent of attempts, nonrecruited participants in 69 percent of trials, rising to 72 percent for nonrecruited participants who saw at least ten videos. The model was right on 80 percent of the 50 videos, well above its 65 percent on the full holdout set, so the sample happened to suit it. Measured one person at a time, 13 percent of recruited participants, 27 percent of nonrecruited participants with at least ten videos and 37 percent of those with fewer than ten beat the model.

03Does a crowd beat the machine?

Averaging the responses per video closes the gap. The crowd mean of recruited participants identified 74 percent of the 50 videos, the crowd mean of nonrecruited participants 80 percent, and the crowd mean of the 1,879 nonrecruited participants who had seen at least ten videos 86 percent, against the model's 80 percent on the same videos. The abstract's summary is that people and the model are "similarly accurate, while making different kinds of mistakes".

Two videos defeated both. The paper describes one as "extremely blurry" and the other as "filmed from a low angle, and the actress's glasses show significant glare". Neither the crowd nor the model had a cue to work with.

04What happens when people see the model's prediction?

The second part of experiment 2 showed recruited participants the model's prediction after their first answer and let them revise. They changed their confidence in 24 percent of trials and crossed the 50 percent line in 12 percent. Accuracy rose from 66 to 73 percent of observations. On the 40 videos the model got right, participants ended 10.4 percent more accurate than before seeing it.

The cost came on the videos the model got wrong. On a deepfake of Kim Jong-un, accuracy fell from 56 to 34 percent after participants saw a confident wrong prediction; on a deepfake of Vladimir Putin it fell from 70 to 55 percent. The abstract states the general finding: "inaccurate model predictions often decrease participants' accuracy". A model shown to a reviewer is a second opinion with the same failure modes as any other, and a persuasive one.

05Where do people and machines fail differently?

The authors hand-labelled seven features of each video: graininess, blurriness, darkness, a flickering face, two people, a floating distraction and dark skin. On the 14 videos that were grainy, blurry or dark, the recruited crowd was right on 8, the nonrecruited crowd and the model on 10; on the other 36 videos the three were right on 29, 32 and 30. A flickering face, a classic generator artefact, raised recruited participants' accuracy by 24.2 percentage points; the kinds of artefact involved are described on the visible signs of a deepfake page.

The perceptual interventions separate the two detectors most clearly. Videos were shown upside down, with the face split horizontally, or with a black bar over the eyes, to disrupt the specialised face processing people rely on. The abstract reports that these manipulations "hinder human participants' performance while mostly not affecting the model's performance": people read a face as a whole, the model reads pixels, and only the first is broken by inversion.

06How do these figures compare with other human tests?

The pattern holds across the row. Köbis and colleagues chose their 16 clips from the 3,000 DFDC videos hardest for classifiers and found 57.6 percent accuracy with a strong bias toward calling videos authentic; neither an awareness message nor a payment improved it (iScience 2021). The FaceForensics++ user study found that people caught DeepFakes and FaceSwap at 76 to 78 percent on raw frames but Face2Face at 49.60 percent and NeuralTextures at 32.28 percent, below chance, because reenactment "does not introduce a strong semantic change" (arXiv 1901.08971). On still synthetic faces people were at chance. Video gives a viewer more to work with than a photograph, and a crowd more again.

07What the numbers do and do not show

Every figure above is a 2020 model on 2020 fakes made from consenting actors under controlled conditions, with no caption, source or context to reason from. The leading model's own 65 percent on the holdout set shows how much harder DFDC was than the curated benchmarks of its day, and detectors have moved on since. What has not moved is the structure of the result: individuals near the model, crowds slightly above it, and the model pulling people down when it is wrong. The same lab-to-world gap applies to voice, where the found-recording audio test reports lab detectors falling toward chance.

None of this measures a live identity check, where a real-time face swap arrives by a route no viewer sees; the virtual camera injection attack page covers that case. For a clip in front of you, the study's practical lesson is the crowd result: several judgements, and a machine score treated as one voice among them rather than the verdict.