Face recognition technology: systems, data and evaluations

This page is the overview for the face recognition section. The face recognition pipeline page goes step by step through the first part; this one places the system, its data and its evaluations side by side.

01What is a face recognition system made of?

A face recognition system does not store faces. It stores feature vectors, which NIST's demographic effects report describes as the output of "a face detector followed by a feature extraction algorithm that converts one or more images of a person into a vector of values that relate to the identity of the person" (NISTIR 8280, 2019). The extractor is a neural network trained on identity-labelled images. Recognition is then "a differential operator": two vectors are compared and a similarity score comes out, which a threshold turns into a match or a non-match.

The same report draws a line that matters for every page in this section. Classification and estimation, such as guessing sex or age from one photo, are "single-shot operations from one sample alone, employing machinery that is different from that used for face recognition". A study of a gender classifier is not a study of a recognition algorithm, and the two are scored with different error rates.

02How is a modern model trained?

The dominant recipe since 2018 is a convolutional network trained with a margin-based softmax loss on millions of labelled web faces. ArcFace, the most cited example, adds an additive angular margin to the softmax so that embeddings of one identity cluster tightly and different identities sit far apart on a hypersphere (Deng et al., arXiv 1801.07698, TPAMI 2021). Its implementation details are typical of the field: face crops normalised to 112 by 112 pixels from five landmarks, ResNet50 or ResNet100 backbones, and a 512-dimensional embedding that takes 8.9 milliseconds per face on ResNet50 and 15.4 on ResNet100 in the paper's measurement.

The loss matters less than the data. CASIA-WebFace, released in 2014, was the first public training set of any size: 10,575 subjects and 494,414 face images collected semi-automatically from the internet, built because "none of large scale face dataset is publicly available" while groups with private data were already reaching 97 to 99 percent on LFW (Yi et al., arXiv 1411.7923). The InsightFace project's dataset list records the sets that followed: MS1M-ArcFace with about 85,000 identities and 5.8 million images, and Glint360K with about 360,000 identities and 17 million images (InsightFace datasets README).

WebFace260M, the benchmark this domain once hosted, is the largest. Its authors downloaded 260 million faces for 4 million names, then ran a self-training cleaning pipeline (CAST) to produce WebFace42M: 2 million identities and 42 million faces, which the paper calls "the largest public face recognition training set" (Zhu et al., arXiv 2103.04098). Trained on it, the authors report a relative 40 percent reduction in failure rate on IJB-C and third place among 430 entries on NIST's FRVT at the time; even a 10 percent subset, WebFace4M, beat the public sets that preceded it.

03Why does training set size matter so much?

Each step up in the figure above changed what a public model could do. The CASIA paper's point in 2014 was that "data is more important than algorithm" for deep face models, and the sets that followed were built to test it. By the time of WebFace42M, the authors could frame the question as closing "the data gap between academia and industry", because commercial vendors had trained on private collections of comparable scale for years.

Size brought two costs. The first is label noise: a name list scraped from the web attaches wrong faces to identities, which is why WebFace260M needed a cleaning stage and why ArcFace's later version added sub-centres per class to tolerate "massive label noise". The second is consent. Every set in the figure was collected from the internet without the subjects' agreement, and several of the sets that preceded WebFace260M were later withdrawn by their publishers.

04How are face recognition systems evaluated?

Academic papers report verification accuracy on fixed pair lists. LFW, with 13,233 images of 5,749 people and 6,000 pairs, is the oldest; ArcFace reports 99.83 percent on it, and the CASIA authors were already quoting 97 to 99 percent from private data in 2014. Harder academic sets such as IJB-C report the true accept rate at a fixed false accept rate instead; WebFace260M's own protocol, FRUITS, adds an inference time budget of 100, 500 or 1,000 milliseconds so that accuracy is reported with its cost.

NIST's Face Recognition Technology Evaluation is the operational counterpart. Developers submit a compiled library, NIST runs it on photographs collected in government applications and publishes the error rates. On the 1:1 track the summary statistic is the false non-match rate at a fixed false match rate, with algorithms ordered by FNMR at FMR of one in a million on the Visa-Border set; the page lists 1,501 algorithms from 461 developers submitted between 2017 and 2026 (NIST FRTE 1:1, read 9 November 2026). The NIST evaluations page explains how to read those tables. The four error rates they use, two for verification and two for identification, are defined on the FMR, FNMR, FPIR and FNIR page.

05Verification versus identification

A verification system answers "is this the same person?" from two photos. An identification system searches one probe against a gallery of N enrolled identities and returns candidates. NISTIR 8280 describes the second as comparing a probe's features "with all feature vectors previously enrolled from 'gallery' images", returning either a fixed number of the most similar candidates or only those above a preset threshold. When the threshold is set to zero, every search returns candidates and the false positive identification rate is 100 percent by construction; that is the investigation mode, where a human reviews the list.

The two modes fail differently and are scored differently. A false positive in verification lets an impostor through; a false positive in identification sends an innocent person's photo to a reviewer. The NISTIR 8280 report found that false positive rates vary across demographic groups "by factors of 10 to beyond 100 times" for many algorithms while false negatives vary "by factors below 3", which is why both rates must be read before a deployment is judged.

06Where face recognition meets deepfakes

The pipeline that identifies a face is also the pipeline that deepfake detectors reuse: the same detector, the same alignment, often the same backbone. That is why detection models borrow recognition's habits and its failure modes. A detector such as GenConViT reports an average AUC of 99.3 percent on the datasets it was trained on and scored 0.63 on fakes collected from the internet in 2024, the same benchmark-to-field gap that recognition met when LFW saturated and NIST's operational tests took over.