AI-generated faces: how StyleGAN and diffusion models make a face

This page is part of the deepfake detection hub. It explains how the faces are made and which data they were learned from; the AI-generated faces page covers how detectors and datasets handle them, and the hub's page on kinds of face deepfake places entire face synthesis next to swaps and reenactment.

01What is an AI-generated face?

A generated face is a picture of a person who does not exist, produced by a network that has learned the statistics of real face photographs. Nothing in it is copied from a single source image: the model has learned what faces look like across its training set and samples a new one from that knowledge. The two families that matter for faces are generative adversarial networks, where StyleGAN and StyleGAN2 set the standard between 2018 and 2020, and diffusion models, which took over from 2021.

That distinction separates this kind of fake from the others on this site. A face swap or a reenactment edits a recording of a real person, so a detector can look for the seam, the mismatch or the blended region. A generated face has no original, so everything a detector or a person can use is a property of the generator itself. The question of how the face was generated therefore comes before the detection question, and this page answers it from the papers that introduced the generators.

02How does StyleGAN make a face?

StyleGAN (Karras, Laine and Aila, 2018) starts from a latent code: 512 numbers drawn at random. A mapping network of 8 fully connected layers turns that code into an intermediate code the authors call w. A synthesis network of 18 layers, two for each resolution from 4 by 4 up to 1,024 by 1,024, then draws the image, starting not from the code but from a learned constant 4 by 4 block. The intermediate code reaches every layer of the synthesis network as a "style" through adaptive instance normalisation, which scales and shifts each feature map, and random noise is added after every convolution to supply the small details (arXiv 1812.04948).

The paper describes the result as an "automatically learned, unsupervised separation of high-level attributes (e.g., pose and identity when trained on human faces) and stochastic variation in the generated images (e.g., freckles, hair)". Coarse layers control pose and face shape, middle layers facial features and hair style, fine layers colour and microstructure. The generator has 26.2 million trainable parameters against 23.1 million for the Progressive GAN generator it replaced, and on the FFHQ dataset it lowered the Fréchet inception distance, the paper's quality measure, from 8.04 to 4.40, where lower is better.

Training is adversarial. A second network, the discriminator, is shown real photographs and generated images and learns to tell them apart; the generator is trained to make images the discriminator accepts. The faces that result are not averages of the training set but samples that the discriminator could not reject.

03What did StyleGAN2 change, and why does it matter for detection?

StyleGAN2 (Karras et al., 2019) opened by diagnosing a flaw in its predecessor: "Instance normalization causes water droplet -like artifacts in StyleGAN images", a blob that appears in the generator's internal feature maps from 64 by 64 resolution onward and is, in the authors' words, "a systemic problem that plagues all StyleGAN images" (arXiv 1912.04958). The paper replaced the normalisation with weight demodulation, added a path length regulariser, and dropped progressive growing. On FFHQ the Fréchet inception distance fell from 4.40 to 2.84.

Two points in that paper bear directly on detection. First, an artefact that was a reliable cue for one generation of models was engineered out in the next; a detector trained on the droplet learned something that stopped existing a year later. Second, the authors themselves wrote that "classifier-based methods can quite reliably detect generated images, regardless of their exact origin", then added that "given the rapid pace of progress in generative methods, this may not be a lasting situation". Frank et al. (ICML 2020) showed where the durable GAN cue lay: in frequency space, GAN images "exhibit severe artifacts that can be easily identified", which the authors found "consistent across different neural network architectures, data sets, and resolutions" and traced to "upsampling operations found in all current GAN architectures" (arXiv 2003.08685).

04How does a diffusion model make a face?

A diffusion model takes the opposite route to a GAN. Ho, Jain and Abbeel (2020) define it as "a parameterized Markov chain trained using variational inference to produce samples matching the data after finite time", trained to reverse a forward process that "gradually adds noise to the data in the opposite direction of sampling until signal is destroyed" (arXiv 2006.11239). In training, real images are corrupted with Gaussian noise over a fixed schedule and the network learns to predict and remove that noise. In sampling, the network starts from pure noise and denoises it step by step until an image remains; the paper used 1,000 steps for every experiment and showed face samples trained on CelebA-HQ at 256 by 256.

There is no discriminator and no single pass from code to image. Because the image is built by many small denoising steps rather than by a stack of upsampling layers, the frequency signature that Frank et al. found in GAN output is not guaranteed to appear in the same form, which is the point at which the detection story on the AI-generated faces page begins. Later text-to-image systems extend the same denoising process, often in a compressed latent space, but the mechanism of adding noise in training and removing it in sampling is the one described in the 2020 paper.

05Which photographs were these faces learned from?

Every generated face is a function of a training set of real ones. StyleGAN introduced Flickr-Faces-HQ (FFHQ), "consisting of 70,000 high-quality images at 1024^2 resolution", and the paper states plainly that "The images were crawled from Flickr". The authors describe the set as offering "vastly more variation" than CelebA-HQ in age, ethnicity, background and accessories such as glasses and hats. The denoising diffusion paper trained its face model on CelebA-HQ, a set of celebrity photographs at 256 by 256.

The provenance of these sets matters twice over. The faces a generator produces inherit the demographic balance and the photographic style of the images it saw, so FFHQ's Flickr portraits set the look of a whole generation of synthetic faces. And the people in those photographs did not sit for a generator; the consent position of web-collected face sets is discussed on the face recognition side of this site, and the same questions apply to the training data behind synthetic faces. The detection datasets that include fully generated faces, and the generators they came from, are listed with the other public sets on the hub's dataset comparison.

06How convincing are the results?

Nightingale and Farid tested exactly this question with StyleGAN2 faces. They took 400 synthetic faces and 400 matched real faces from FFHQ and asked 315 participants to classify 128 of them each as real or synthesized. The average accuracy was 48.2 percent, with a 95 percent confidence interval of 47.1 to 49.2 percent, against chance performance of 50 percent (doi 10.1073/pnas.2120481119). A second experiment with 219 new participants, given training and trial-by-trial feedback, reached 59.0 percent, and the authors report there was no improvement in accuracy over time. In a third experiment 223 participants rated trustworthiness on a 1 to 7 scale: real faces averaged 4.48 and synthetic faces 4.82.

Those are the numbers behind the claim that StyleGAN2 faces pass as real. They were measured on still images from one generator in 2021 to 2022, with faces matched for the comparison, and they are the figures the can people spot AI faces page carries alongside the deepfake video studies.

07What this means for detection and for provenance

A generated face carries no trace of an edit, only traces of its generator, and the two generator families leave different traces. StyleGAN's droplet was removed within a year; the GAN frequency signature does not transfer unchanged to diffusion output. That is why generated-face detectors are reported per generator family in the papers, and why a score on one family says little about the next. The classifiers that do the scoring are, for the most part, the same image networks used elsewhere in face analysis, as the face recognition backbones in detectors page shows for DeepfakeBench; the numbers they have reached on generated faces, and the datasets built to test them, are on the AI-generated faces page.

Because a synthetic face has no original to compare against, the question "where did this file come from?" does more work than "does it look real?". Provenance records such as Content Credentials (C2PA) can state that an image was created by a generator, when the generating software signs that fact into the file, and the deepfake statistics with a primary source page lists the human and detector figures for generated faces that trace back to a paper.