Face recognition accuracy: how it is measured and what NIST found

This page sits in the explainer part of the face recognition section. The face verification metrics page defines the error rates; this page explains why a single percentage is never an accuracy figure and what the largest evaluation found when it measured accuracy properly.

01Why is face recognition accuracy not a single percentage?

A face recognition algorithm never decides anything about one image alone. It compares two feature vectors and emits a similarity score, and a threshold on that score turns the comparison into a match or a non-match. Two mistakes are then possible: a false match, when two different people score above the threshold, and a false non-match, when two images of one person score below it. Raising the threshold trades one for the other, so any algorithm can show a false match rate near zero if a high false non-match rate is accepted, and the reverse.

A percentage therefore describes an operating point, not an algorithm. The convention NIST uses fixes the false match rate first and reports the false non-match rate at that threshold, ordering its 1:1 table by FNMR at an FMR of one in a million on the VISABORDER set (NIST FRTE 1:1). NISTIR 8280 notes that operations target false match rates "spanning several decades 10^-6 to as high as 10^-2", so the same algorithm has a different FNMR at each (NISTIR 8280, section 3). The four error rates page gives the formal definitions, including the one-to-many pair FPIR and FNIR.

02What does a published accuracy figure look like?

Three figures from the literature show the form. The Partial FC paper reports that its best configuration reaches "97.2% TAR on IJB-C at FAR=10^-4", a true accept rate at a false accept rate of one in ten thousand on a named set (arXiv 2010.05222). The ArcFace paper reports 99.83 percent verification accuracy on LFW for a ResNet100 trained on MS1MV3, a pair-accuracy protocol on 6,000 fixed pairs on which groups with private training data were already reporting 97 to 99 percent in 2014, as the CASIA-WebFace paper recorded (arXiv 1411.7923). The WebFace260M paper reports that training on its cleaned 42 million faces reduced the relative failure rate on IJB-C by 40 percent and ranked third of 430 entries on NIST's evaluation at the time.

Each figure in the table carries its metric, its operating point and its dataset; none of them can be compared with another row, because the sets and the operating points differ. A reader who meets "99 percent accurate" without those three facts has not been given an accuracy figure.

03How does NIST measure face recognition accuracy?

NIST's Face Recognition Technology Evaluation, long known as FRVT, tests algorithms that developers submit as compiled software on image sets the developers never see, with results updated as submissions arrive and participation open "no more frequently than four calendar months" per developer. Its 1:1 page reports FNMR at specified FMR values across several datasets, and its 1:N page reports FNIR at the threshold that limits FPIR to 0.003, "the proportion of non-mated searches producing one or more candidates above threshold" (NIST FRTE 1:N). The NIST FRTE page describes how to read those tables.

The demographic study used the same machinery at scale. NISTIR 8280, published 19 December 2019, processed "18.27 million images of 8.49 million people through 189 mostly commercial algorithms from 99 developers" on four datasets: domestic mugshots, immigration application photographs, visa photographs and border crossing photographs. The report's method point is that "the vast majority of biometric systems are configured with a fixed threshold against which all comparisons are made", so demographic effects must be read at a fixed threshold; reporting false negative rates at fixed false positive rates per group hides the false positive differentials.

04What did NIST find about accuracy across demographic groups?

The main result is that "false positive differentials are much larger than those related to false negatives and exist broadly, across many, but not all, algorithms tested". Across demographics, "false positives rates often vary by factors of 10 to beyond 100 times", while false negatives "tend to be more algorithm-specific, and vary often by factors below 3". With application photographs, false positives were highest in West and East African and East Asian people and lowest in Eastern European individuals, and for a number of algorithms developed in China the effect was reversed, with low false positive rates on East Asian faces. False positives were higher in women than men, an effect smaller than that due to race, and elevated in the elderly and in children.

Two qualifications in the report matter for reading any accuracy claim. Some developers "supplied highly accurate identification algorithms for which false positive differentials are undetectable", so the differential is a property of particular algorithms, not of the technology. And the report distinguishes face recognition from face analysis: the gender classification studies often cited for bias "did not evaluate face recognition algorithms", and NIST recommends that reports of bias state the class of algorithm evaluated. The demographic differentials page reads these findings in full.

05How should a reader check an accuracy claim?

Ask four questions of any figure. Which metric: FNMR, FMR, TAR, FAR, FPIR, FNIR or a pair accuracy? Which operating point: the FMR or FAR target, or the threshold, at which it was read? Which dataset, and was the algorithm's developer able to see it? And, for a deployment, what are both error rates for each demographic group at the threshold actually used, which NISTIR 8280 says "is rarely done". A figure that answers all four can be compared with another that answers them the same way. A figure that answers none of them is a marketing number.

The same discipline applies to the deepfake side of this site, where a detector's AUC on the fakes it trained on is quoted as if it were its accuracy in general. The accuracy versus AUC page shows what a threshold and a class balance do to a detection percentage, and the published detector ranges for seen, unseen and wild fakes show how far the three conditions sit apart. No single benchmark score predicts how a detector performs on unseen manipulations, and no single percentage describes a face recognition algorithm.