Liveness detection datasets: OULU-NPU, CASIA, SiW, CelebA-Spoof

This page belongs to the deepfake detection hub and sits beside the face anti-spoofing guide, which covers methods and error rates. Here the subject is the data: which datasets exist, how large they are, what attacks they hold and how a paper should test on them.

01What is a liveness detection dataset?

A liveness detection dataset is a labelled collection of genuine face captures and attack captures, built to train and test software that decides whether a face in front of a camera is a live person. ISO/IEC 30107-1:2023 calls an attack a "presentation to the biometric capture subsystem with the goal of interfering with the operation of the biometric system", and genuine captures "bona-fide presentations". Datasets in this field hold both, and label the attack type, so error rates can be reported per attack.

Papers in the field call the task face anti-spoofing. The liveness detection overview explains how the standards vocabulary relates liveness checks to presentation attack detection.

02Which datasets do papers use, and how big are they?

Six datasets recur in the papers this page draws on. Counts come from each dataset's own page or paper where one could be read, and otherwise from the comparison table in the SiW paper (Liu, Jourabloo and Liu, CVPR 2018, Table 1).

The SiW table lists the first dataset as CASIA-MFSD and cites the 2012 ICB paper "A face antispoofing database with diverse attacks", which is the CASIA-FASD paper. The three oldest sets are small and frontal, with no more than two replay devices each, and the SiW paper uses two of them, CASIA-MFSD and Replay-Attack, as its cross-dataset test pair.

03Why do dataset sizes differ between sources?

OULU-NPU is the clearest case. The project page says the database "consists of 4950 real access and attack videos". The SiW paper's comparison table lists 1,980 live and 3,960 attack videos for the same dataset, which adds to a different total. The two sources count differently and neither explains the gap, so this page uses the project page figure and flags the other.

The rule for a citation is the same as elsewhere on this site: quote the count from the dataset owner, name the page, and say what the count covers. A subject count is safer than a video count, because 55 subjects for OULU-NPU appears in both sources.

04How do the four OULU-NPU protocols work?

The 55 subjects are split into three subject-disjoint groups for training, development and testing, so no person appears in two groups. Each of four protocols then withholds one source of variation.

• Protocol I tests previously unseen illumination and background scene, using the three recording sessions.
• Protocol II tests a previously unseen print and video-replay attack, so the test attacks use a printer and display the training set never saw.
• Protocol III is leave one camera out: videos from five phones train the model and the sixth phone tests it, repeated for each phone.
• Protocol IV combines all three, and the project page calls it "the most challenging protocol".

The dataset is released under a signed end user licence agreement that must come from a person with a permanent position at an institution. The project page includes a baseline, a colour texture method that computes local binary pattern histograms in two colour spaces and classifies ten frames per video.

05What do SiW and CelebA-Spoof add?

SiW, Spoof in the Wild, has 165 subjects with 8 live and 20 attack videos each, in 4,620 videos. Its live videos come from four sessions that vary head pose, expression and lighting, with a yaw range of minus 90 to 90 degrees, where the older sets are frontal. Its attacks are two print and four replay types. The paper also defines three protocols, including one that trains on print and tests on replay and the reverse.

CelebA-Spoof is the largest by an order of magnitude: 625,537 pictures of 10,177 subjects, captured in 8 scenes (2 environments and 4 illumination conditions) with more than 10 sensors. It carries 10 spoof type annotations and the 40 attribute annotations of the CelebA dataset it is built on, according to its ECCV 2020 abstract. It is an image set, so methods that use video cues such as heartbeat signals cannot train on it.

06How much does the score drop on a different dataset?

Cross-testing, training on one dataset and testing on another, is the common check. In the SiW paper's Table 6, a colour texture method scored a half total error rate of 30.3 percent when trained on CASIA-MFSD and tested on Replay-Attack, and 37.7 percent the other way. The authors' depth and rPPG method scored 27.6 and 28.4 percent, the best of the methods compared. Half total error rate is the mean of the false rejection and false acceptance rates, so 28 percent means roughly one error in four presentations.

Within a dataset the same method does far better: on OULU-NPU Protocol I it reports an ACER of 1.6 percent, and on Protocol IV 9.5 plus or minus 6.0 percent. The APCER, BPCER and ACER page defines the three rates. The pattern matches face deepfake detection, covered under cross-dataset generalisation: a score is a statement about one dataset and one protocol.

07Which attacks do these datasets leave out?

OULU-NPU holds print and video-replay attacks, and SiW holds print and replay attacks. Neither describes an injected video, a 3D mask or a makeup attack, so an error rate measured on them says nothing about those attack types. Injection, where manipulated video enters the system without passing through the camera at all, is outside what a presented-copy dataset can test; the wider deepfake dataset catalogue lists sets built for manipulated video instead.

A fair reading of a liveness paper therefore asks four things: which dataset, which protocol, which attack types appear in the test set, and whether the method saw those types in training. The same questions apply to fingerprint and iris data, and speech is covered under voice liveness detection. The page on liveness checks across fingerprint, iris and voice compares.