Face anti-spoofing: datasets, protocols and published error rates

This page is part of the deepfake detection hub, in the liveness and identity-check corner that the liveness and injection attacks page introduces. That page covers the threat model; this one covers the datasets, the protocols and the numbers.

01What is face anti-spoofing?

The standards vocabulary calls the task presentation attack detection, PAD. ISO/IEC 30107-1:2023 defines PAD as "automated discrimination between bona-fide presentations and biometric presentation attacks", and treats liveness detection as one family of PAD methods rather than a synonym. The attacks in scope are those made at the capture device: a printed photograph held to the camera, a video replayed on a screen, a mask. A deepfake injected into the video stream after the camera is a different attack class, covered on the presentation attack detection page.

The research literature kept the older name. Face anti-spoofing papers cast the problem as a binary classification of live against spoof, and since about 2017 the better ones add auxiliary signals. The SiW paper trains a network to estimate the depth map of the face, which is flat for a print or a screen, and the remote photoplethysmography signal, the small colour change of skin with each heartbeat, which a replay cannot reproduce in sync (arXiv 1803.11097). Both are properties of a live face that a presented copy lacks, which is the shape of every anti-spoofing cue.

02Which datasets are used, and how big are they?

Three public datasets carry most of the published results. Older sets, including CASIA-FASD (2012), Replay-Attack (2012) and MSU-MFSD (2015), are listed in the SiW paper's comparison table with their counts; they are smaller, frontal and recorded with one or two devices, and remain in use for cross-dataset tests.

OULU-NPU was built at the University of Oulu and Northwestern Polytechnical University to test "generalization performances of face PAD techniques in mobile scenarios", and is distributed under a signed licence from an institutional address. SiW, Spoof in the Wild, was recorded with head motion, expression changes and a moving light source, which the frontal sets lacked. CelebA-Spoof is two orders of magnitude larger in subjects because it is built on the CelebA identities: its authors describe 625,537 pictures of 10,177 subjects, spoof images "captured from 8 scenes (2 environments * 4 illumination conditions) with more than 10 sensors", and 10 spoof type labels on top of the 40 CelebA attributes (arXiv 2007.12342). The SiW paper's comparison table gives OULU-NPU as 1,980 live and 3,960 attack videos, which sums to 5,940; the project page's figure of 4,950 is used here because it is the dataset's own.

03What do the four OULU-NPU protocols hold out?

Protocols are what make an anti-spoofing result mean something, because a detector that has seen the test camera, the test printer and the test room is being asked a much easier question than one that has not. OULU-NPU splits its 55 subjects into three subject-disjoint sets for training, development and testing, and then defines four protocols on top.

Protocol I holds out the environment: training and test videos come from different sessions, so illumination and background are unseen. Protocol II holds out the attack instruments, with "a previously unseen print and video-replay attack in the test set" made on the second printer and display. Protocol III holds out the camera in a leave-one-camera-out scheme: five of the six phones train the model and the sixth tests it, repeated six times, which is why its results are reported with a standard deviation. Protocol IV, in the project page's words, is "the last and most challenging protocol" where "all above three factors are considered simultaneously". The protocol ladder is the closest thing anti-spoofing has to the cross-dataset tests used for face deepfake detectors.

04How are results reported: APCER, BPCER and ACER?

The two error rates come from ISO/IEC 30107-3:2023, the testing and reporting part of the PAD standard. As the SC37 working group's own slides quote it, APCER is the "proportion of attack presentations using the same PAI species incorrectly classified as bona fide presentations in a specific scenario", and BPCER is the "proportion of bona fide presentations incorrectly classified as attack presentations in a specific scenario". APCER is the security error, an attack let through. BPCER is the convenience error, a real person rejected. Both are defined per attack type, the PAI species, so a print APCER and a replay APCER are separate numbers.

Papers usually add ACER, the average of APCER and BPCER, and report it with the two components for each protocol. The standard itself points the other way: the same slides record its advice against single figures such as the equal error rate or the half total error rate, and its convention of reporting BPCER at a fixed APCER, written BPCER20 at 5 percent, BPCER100 at 1 percent and BPCER10 at 10 percent. A paper's ACER and a certification lab's BPCER20 are therefore not the same quantity, and a low ACER can hide a high BPCER at the APCER a deployment would accept.

05What error rates have been published?

The SiW paper reports its depth and rPPG method on all four OULU-NPU protocols alongside the entries from the 2017 competition on the same data. On Protocol I it reaches an APCER of 1.6 percent, a BPCER of 1.6 and an ACER of 1.6; on Protocol II, 2.7, 2.7 and 2.7; on Protocol III, 2.7 plus or minus 1.3, 3.1 plus or minus 1.7 and 2.9 plus or minus 1.5 across the six held-out cameras; and on Protocol IV, 9.3 plus or minus 5.6, 10.4 plus or minus 6.0 and 9.5 plus or minus 6.0. The competition entry GRADIANT, the strongest comparison on the hardest protocol, scored an ACER of 10.0 plus or minus 5.0 on Protocol IV. The authors note that their method "achieves the lowest ACER in 3 out of 4 protocols".

On its own three SiW protocols the same method reports ACERs of 3.58 percent (unseen pose and expression), 0.57 plus or minus 0.69 (unseen replay medium) and 8.31 plus or minus 3.81 (train on print, test on replay, and the reverse). The third figure is the one to remember: a detector trained on one attack type and tested on another loses more than a detector tested on an unseen pose or an unseen screen.

06Why does Protocol IV cost six times the error of Protocol I?

The same method, the same data and the same metric give 1.6 percent on Protocol I and 9.5 percent on Protocol IV. The difference is entirely in what the detector has not seen: a new room, a new printer and display, and a new phone camera, together. Each factor changes the pixels a spoof leaves behind. Print artefacts depend on the printer's dot pattern, replay artefacts on the display's pixel grid and refresh, and both are filtered through the sensor and compression of the recording phone. A detector trained on five phones has learned the noise of five sensors, and the sixth is a distribution it has never seen.

The pattern matches what the rest of this site records for other detectors. Face deepfake detectors lose a tenth or more of their AUC when the encoder changes, as the compression and deepfake detection page shows, and speech spoofing detectors in the ASVspoof challenge series lose generalisation across source datasets. No single benchmark score predicts how a detector performs on unseen manipulations, and in anti-spoofing the protocol number next to the score says how much was unseen.

07Where anti-spoofing sits in a deployed identity check

A remote identity check runs anti-spoofing and face recognition on the same frames. The recognition step asks whether the face matches the document; the anti-spoofing step asks whether there is a live face at all. The error rates of the two steps are measured separately, by different standards and different laboratories, and a system's published recognition accuracy says nothing about its APCER. The face recognition system components page sets out every stage of that pipeline and which stage each published error rate describes.

What the datasets above do not cover is also worth stating. OULU-NPU and SiW test print and replay attacks presented to a camera. None of the three sets contains an injected deepfake video, a silicone mask or a makeup attack, and a BPCER or APCER measured on them describes those two attack species and no others.