AI face recognition: what the deep learning era changed

This page is part of the face recognition section. The how face recognition works page walks through the four steps of the pipeline; this page is about what the deep learning era changed in the data, the models and the benchmarks.

01What changed when deep learning arrived?

The step that changed is the third one, the embedding. Before learned embeddings a face was described by features an engineer designed, and two faces were compared on those. FaceNet, published in 2015, trained a deep convolutional network to map a face directly to a point in a 128-byte Euclidean space "where distances directly correspond to a measure of face similarity", using triplets of matching and non-matching faces chosen online during training (arXiv 1503.03832). Verification, identification and clustering all became operations on distances between points.

ArcFace, from 2018 and presented at CVPR 2019, kept the idea and changed the loss. It normalises the features and the class weights and adds an angular margin of 0.5 to the target logit, producing a 512-dimensional embedding from a 112 by 112 crop aligned on five facial points (arXiv 1801.07698). With a ResNet100 backbone trained on MS1MV2 it reported an accuracy of 99.83 percent on LFW and a true accept rate of 95.6 percent at a false accept rate of one in ten thousand on IJB-C. The face recognition technology overview describes how such a model is trained and deployed.

02How did the training sets grow?

Embeddings are only as good as the identities they are trained to separate, and the era is a story of data. FaceNet's authors trained on a private set of "between 100M-200M training face thumbnails consisting of about 8M different identities" and reported, in section 5.5 of the paper, that tens of millions of images cut error by 60 percent relative to millions, while hundreds of millions gave a boost that "tapers off". No one outside the authors' organisation could reproduce the result, so the research field built its own sets from the web.

MS-Celeb-1M released 10 million images of celebrities in 2016 and is no longer served by its authors; ArcFace trained on MS1MV2, a cleaned subset of 85,000 identities and 5.8 million images. Glint360K followed with 360,000 identities and 17 million images. WebFace260M collected 4 million identities and 260 million faces and cleaned them with a self-training pipeline called CAST into WebFace42M, 2 million identities and 42 million faces (arXiv 2103.04098). Its authors reported that even a tenth of the data outperformed the public training sets of the time, and that the full WebFace42M reduced the relative failure rate on IJB-C by 40 percent. The rights position of each of these sets is covered on the face datasets and consent page of this section.

03Why did LFW stop being the benchmark?

Labeled Faces in the Wild holds 13,233 images of 5,749 identities, and from 2007 it was the standard verification test. FaceNet's 99.63 percent in 2015 cut the error against the best published result by 30 percent; ArcFace's 99.83 percent in 2019 moved it by a further fifth of a point. Once the leading models all sit within a fraction of a point of each other and of the ceiling, the test no longer separates them, and the remaining errors are as likely to be label noise as model failure.

MegaFace tried to restore difficulty by adding one million distractor photographs of 690,572 people to the gallery, so that a probe had to find its mate among strangers. Its organisers retired it after models reached "benchmark performance of over 99%", and the dataset is no longer distributed (megaface.cs.washington.edu). Saturation, not a change of fashion, is why the field moved its headline number.

04What replaced it: TAR at FAR on IJB-C

The IARPA Janus Benchmark C, IJB-C, holds 3,531 subjects in 21,294 images and 117,542 frames from 11,779 videos. Its verification protocol reports the true accept rate at a fixed false accept rate, usually one in ten thousand or one in a million, so a model is judged by how many genuine pairs it accepts while held to a fixed rate of impostor acceptance. The face verification metrics page defines TAR, FAR and the ROC curve behind them.

The numbers on that scale still move where LFW does not. ArcFace reported 95.6 percent TAR at a FAR of one in ten thousand; Glint360K's authors reported 97.2 percent at the same operating point (arXiv 2010.05222); WebFace42M's 40 percent relative reduction in failure rate is a statement about the same test. NIST, which distributed the Janus sets, discontinued all three IJB challenges on 14 March 2023, so new results on IJB-C come from groups that already held the data.

05How does NIST test operationally?

The second move was from academic sets to operational images. NIST's Face Recognition Technology Evaluation runs continuously on photographs of the kind an agency actually holds, and reports, for the 1:1 track, the false non-match rate "set to achieve the false match rate (FMR) specified", and for the 1:N track the false negative identification rate at a false positive identification rate of 0.003 (NIST FRTE 1:1). Results are updated monthly and a developer may submit no more than once every four months. The NIST evaluations page explains how to read the report pages.

That track is where the data race was finally scored. The WebFace260M paper reports that a model trained on its data ranked third among 430 entries on the NIST evaluation at the time. The same programme also produced the era's most cited caveat: NISTIR 8280, in December 2019, ran 189 algorithms over 18.27 million images of 8.49 million people and found false positive rates that "often vary by factors of 10 to beyond 100 times" across demographic groups, while false negative differences were usually below a factor of 3. Accuracy on a saturated benchmark says nothing about either figure.

06What the deep learning era did not change

The pipeline is the same four steps it was before deep learning: detect, align, embed, compare. What changed is that the third step is learned and the first and fourth are measured differently. The errors are still false matches and false non-matches, counted at a fixed threshold, and a verification score is still only meaningful next to a second face; the FMR, FNMR, FPIR and FNIR page of this section keeps the four definitions in one place. A match score, like a deepfake detector score, is inferred from pixels rather than read from a record, the distinction the provenance or detection page draws.

Nor did the era settle the data question. The sets that made the embeddings possible were collected from the web without the subjects' consent, and two of the largest, MS-Celeb-1M and MegaFace, have been withdrawn; the IJB sets are no longer distributed either. A face recognition model from 2019 is described by its backbone, its loss, its training set and its TAR at a stated FAR on a stated test, and a reader who is given fewer than those four facts has not been given a result.