Adversarial attacks on deepfake detectors: what the papers show

This page is part of the deepfake detection hub and is the third of three generalisation failures the site documents: the cross-dataset generalisation drop happens with no attacker, the detection in the wild drop happens with an unknown generator, and the drop here happens when someone is trying.

01What is an adversarial attack on a detector?

A detector is a function from an image to a score. An adversarial example is an input chosen to make that score wrong, built by adding a perturbation small enough that the image still looks the same. Carlini and Farid bound the perturbation by a pixel norm and solve for the smallest change that flips the decision; Hussain et al. bound each pixel's change by a maximum of 16/255 on a 0 to 1 scale and use gradient sign methods, because a video has thousands of frames and each needs its own perturbation (arXiv 2002.12749).

Two threat models recur in both papers. In a white-box attack the attacker has the detector's weights and can compute gradients through it. In a black-box attack the attacker can only submit inputs and read scores, or has no access at all and must attack a stand-in model and hope the perturbation transfers. The second is the realistic case for a detector run as a service, and both papers report it separately.

02What did Carlini and Farid measure on image detectors?

The target was the CVPR 2020 classifier of Wang et al., a ResNet-50 trained on ProGAN images that generalises to ten other generators and, on the 94,036-image dataset its authors released, reaches an AUC of 0.97 (arXiv 2004.00622). Carlini and Farid ran five attacks on a random subset of 10,000 fake images at 224 by 224 pixels. Flipping only the lowest-order bit of pixels drove the AUC to 0.0005; at a fixed 5 percent false positive rate, an L2 distortion of 0.02 brought the true positive rate to chance and 0.1 brought it to 0.1 percent, a distortion the authors note is over 16 times smaller than what reduces an ImageNet classifier of the same architecture to chance.

A single universal patch overlaid on any fake image reduced the AUC from 0.966 to 0.085, and a single adversarial direction added in StyleGAN's latent space, so that the generator itself produced evasive faces, reduced it from 0.99 to 0.17. The black-box attack trained a separate ResNet-18 with an AUC of 0.85, built adversarial examples against it, and transferred them: the target's AUC fell from 0.96 to 0.22, below chance. The same lowest-bit attack on the Frank et al. frequency classifier took its true positive rate on ProGAN images from 99 percent to 0 percent with half the pixels flipped. The authors withheld their code and note that a defender "has to prevent all possible attacks" while the attacker needs one.

03What did Hussain et al. measure on video detectors?

The targets were XceptionNet and MesoNet trained on FaceForensics++ HQ, where they score 97.49 and 89.55 percent accuracy on the Deepfakes subset before any attack (the paper's Table 1, citing the FaceForensics++ results). The attack success rate is the share of adversarial frames the detector labels real, reported on uncompressed frames (SR-U) and after MJPEG compression (SR-C), averaged over the four FaceForensics++ manipulations.

All figures are percentages from Tables 2, 4, 6 and 7 of the paper. The black-box attacks estimated gradients by querying the model, at an average of 985 queries per frame for the plain version and 2,153 for the robust one. The paper also attacked a 3D CNN sequence detector on DFDC validation videos (Table 8): 91.74 percent accuracy with no attack, a robust white-box success rate of 100 percent on compressed video, and black-box success rates of 24.43 and 51.02 percent. The video-level detection page describes what such sequence models look at.

04Why does compression matter to the attacker?

The first column of the table shows a near-total white-box success, and the second shows what MJPEG compression does to it: XceptionNet's success rate falls from 99.85 to 58.46 percent because re-encoding destroys part of a perturbation tuned to exact pixel values. Hussain et al. answer with a "robust" attack that optimises the perturbation over a distribution of transformations, twelve functions drawn at random from four families (Gaussian blur, Gaussian noise, translation, and downsizing followed by upsizing), and the success rate on compressed video returns to 98.23 percent. The paper's prose gives 98.07 and 99.83 percent for the two robust white-box rows where its Table 4 prints 98.23 and 99.50; this page uses the table. Under H.264 at the FaceForensics++ heavy compression setting (constant rate factor 40) the robust attack still passed 80.39 percent of XceptionNet frames and 90.50 percent of MesoNet frames.

The compression and deepfake detection page documents the better-known effect, that compression lowers detector accuracy on ordinary fakes. The two effects pull in opposite directions: compression that strips a naive adversarial perturbation also strips the artefacts a detector uses, and an attacker who plans for compression loses little.

05What do detector authors say about their own detectors?

The attack papers are not alone in the record. The Face X-ray authors list adversarial examples among the limitations of their own method in section 6 of their paper, alongside its failure on fully synthetic images and its drop to an AUC of 61.6 percent on heavily compressed FaceForensics++ (arXiv 1912.13458). Hussain et al. close by recommending adversarial training and evaluation "under different attack scenarios and attacker capabilities", a protocol that none of the benchmarks whose tables this site reproduces yet includes. The audio field has moved first: ASVspoof 5 incorporated "adversarial attacks for the first time" among its 32 attack types in 2024.

06How to read a detector score after this

A published AUC describes a detector against the fakes in a test set, made by people who were not trying to evade it. Every figure on this page describes the same kind of detector against fakes made by people who were. The honest summary of the two papers is narrow: on the detectors and sets each names, a white-box attacker wins almost completely, a black-box attacker wins most of the time, and compression is a partial defence only against an attacker who ignored it. No single benchmark score predicts how a detector performs against an adaptive adversary, and no detector on this site reports one.

The same discipline applies beyond saved clips. The real-time video call deepfakes page records the one challenge-response test built around an imposter who adapts, and the facial recognition privacy page records a regulator finding against a face system whose operator "failed to take reasonable measures" and never measured its false positives before deployment. A detector deployed without an adversarial evaluation is in the same position.