Human or AI? What the studies measured about telling them apart

This page is part of the deepfake detection hub. It collects the controlled studies behind the question and states their conditions; the human studies page covers the "which face is real" format in more depth, and the AI-generated faces page describes the generators the still-image studies used.

01What the question covers and what this page answers

"Human or AI" is asked of three things: a face image, a video of a person speaking, and a passage of text. The published, peer-reviewed measurements this site can cite cover the first two. For text there is no benchmark on this site and no study is summarised here, so nothing on this page should be read as a statement about written content. For faces and face video the question has been measured in three studies with pre-registered or published protocols, and their numbers are the subject of the sections below.

Each study reports a percentage of correct judgements against a chance level of 50 percent, because every task presented equal numbers of real and synthetic items. The percentages are not comparable across studies without their conditions: the items differ (a generated still face against a face-swapped video), the task differs (one item at a time against a pair), and the viewers differ (open web visitors against paid participants).

02How well do people tell a real face from a generated one?

Nightingale and Farid built a set of 400 faces generated by StyleGAN2 and 400 real faces from the FFHQ photographs the generator was trained on, each real face matched to a synthetic one for gender, age, race and overall appearance (doi 10.1073/pnas.2120481119). In experiment 1, 315 participants each classified 128 of those faces, one at a time, as real or synthesized. Their average accuracy was 48.2 percent, with a 95 percent confidence interval from 47.1 to 49.2 percent, which the authors describe as "close to chance performance of 50%" with no response bias.

Experiment 2 gave 219 new participants a short training and trial-by-trial feedback. Accuracy rose to 59.0 percent (57.7 to 60.4), and the paper notes that "there was no improvement in accuracy over time" within the session. In experiment 3, a further 223 participants rated trustworthiness on a 1 to 7 scale and gave synthetic faces 4.82 against 4.48 for real ones. For both real and synthetic images, White faces were the least accurately classified, which the authors hypothesise is because White faces are overrepresented in the StyleGAN2 training data and so come out more realistic.

03How well do people spot a deepfake video?

Two studies used clips from the Deepfake Detection Challenge set, where paid actors were face-swapped with several methods. Köbis, Doležalová and Soraperra (iScience 2021) ran a pre-registered experiment with 210 participants recruited on Prolific, 83.8 percent of them in the United Kingdom, each shown 16 clips of about 10 seconds drawn from the 3,000 hardest DFDC videos and asked whether each was a deepfake (doi 10.1016/j.isci.2021.103364). Overall accuracy was 57.6 percent, about 9 of 16 correct, and neither an appeal to the importance of the task nor a payment per correct answer changed it.

Groh, Epstein, Firestone and Picard (PNAS 2022) ran two online studies with 15,016 participants in all (doi 10.1073/pnas.2110013119). Experiment 1 showed 5,524 visitors pairs of videos, one real and one fake, and asked which was the deepfake; 82 percent of those who saw at least ten pairs outperformed the leading challenge model's 65 percent holdout accuracy, though the authors leave the direct comparison of people and model to experiment 2, where both judge the same holdout clips. Experiment 2 showed single videos to 9,492 participants: the 304 recruited participants identified the fakes correctly in 66 percent of attempts, the organic visitors in 69 percent of trials, and 72 percent among visitors who saw ten or more videos.

04Are people better or worse than a detector?

On the same 50 holdout videos, the leading DFDC model was correct on 80 percent, and Groh et al. report that this is well above the 65 percent it scored on the full 4,000-video holdout set. Individual viewers were below it; the crowd was not. Averaging viewers' responses per video, the crowd mean reached 74 percent for recruited participants, 80 percent for organic visitors and 86 percent for the 1,879 visitors who had seen at least ten videos. The abstract's phrasing is the one to carry: humans and the model were "similarly accurate, while making different kinds of mistakes".

Showing viewers the model's prediction helped on average and hurt on specific clips. Recruited participants rose from 66 to 73 percent when they could see the model's output, but on two clips where the model was confidently wrong their accuracy fell, from 56 to 34 percent on one and from 70 to 55 percent on the other. The paper also found that presenting videos upside down, with the face split and misaligned, or with a bar over the eyes hurt people's accuracy while "mostly not affecting the model's performance", which the authors read as evidence that human judgements draw on face-specific perception.

05Why do people miss deepfakes?

Köbis et al. report a direction to the errors. Although participants were told that half of the clips were fake, they answered "authentic" 67.4 percent of the time, and for most of the deepfake clips they were correct at below-chance rates. The authors call this a "seeing-is-believing" heuristic. They also found that participants overestimated their own performance, which they name as the second way people are fooled.

The still-image result points to a different cause. Nightingale and Farid's participants showed no response bias at all; they were simply unable to separate the two sets, and feedback moved them only to 59 percent. The generators had, in the authors' words, "passed through the uncanny valley". A face-swapped video leaves more traces than a generated still, which is one reading of why the video studies sit above chance and the image study does not.

06How to use these numbers

A single accuracy figure for "humans versus AI" does not exist; there are at least five, and each belongs to one task. The stills result is 48.2 percent for untrained viewers of StyleGAN2 faces in 2022. The video results are 57.6 percent for one viewer on hard DFDC clips, 66 to 72 percent for one viewer on random holdout clips, and 80 to 86 percent for a crowd. Each comes with a date, and the generators have changed since. Detectors have the same problem in a different form: the cross-dataset drop by detector page lists how much they lose on fakes they were not trained on, and the deepfake voice page has the audio equivalent.

The quotable caveat is the one all three studies support: no measured human accuracy on faces or video transfers to a new generator, a new platform or a different medium. For a live identity check, the question is not asked of a viewer at all; the liveness detection page covers the tests that decide whether a real person is present at the camera.