Face recognition testing: protocols, datasets and reports

This page belongs to the face recognition section and explains the test behind every figure the section quotes. The face verification metrics page defines the error rates themselves; this one covers how a test is set up and reported.

01What does a face recognition test measure?

A test measures how often a system makes each of two errors at a chosen operating point. In one to one verification, a false non-match is a failure to match two images of the same person and a false match is a match between images of different people. NIST's 1:1 page states the definitions it reports: "FNMR is the proportion of mated comparisons below a threshold set to achieve the false match rate (FMR) specified. FMR is the proportion of impostor comparisons at or above that threshold" (NIST FRTE 1:1).

In one to many identification the probe is searched against a gallery and the two errors change name. NIST's 1:N page: "FNIR is the proportion of mated searches failing to return the mate above threshold. FPIR is the proportion of non-mated searches producing one or more candidates above threshold" (NIST FRTE 1:N). The 1:1 versus 1:N face recognition page explains why a gallery of N identities multiplies the chances of a false positive, which is why the two tasks are reported apart. NISTIR 8280 notes that ISO/IEC 19795-1 "requires different metrics to be reported for identification and verification implementations".

02Which datasets are used, and what do they contain?

The dataset decides who is in the test and how many comparisons it can support. Three sets stand behind most academic figures, and their counts are small next to the training sets that produce the models.

LFW's 6,000 pairs made it the first common verification protocol, and it saturated: FaceNet reported 99.63 percent on it in 2015 and ArcFace 99.83 percent in 2019, a gap too small to separate current systems. IJB-C added video frames and a template protocol, in which several images of one person are pooled into one template before matching, and its Test 1 pairs 19,557 genuine matches against 15,638,932 impostor ones so that false match rates near one in ten thousand can be measured (IJB-C README, NIST, December 2017). The list of face recognition benchmarks covers the other public sets with their counts.

03How is the protocol fixed?

The protocol says which comparisons are made and how the threshold is set. A fixed pair list, as in LFW, means every system scores the same 6,000 decisions and reports accuracy on them. A template protocol, as in IJB-C, states how images are grouped before matching. A sequestered protocol, as at NIST, keeps the images from the developer entirely: the algorithm is submitted as a library and run by the testing body on photographs it never sees.

The threshold policy is the part most often dropped from a quoted number. NIST's 1:1 table orders algorithms by FNMR at FMR 0.000001, and its demographic tables fix the threshold for each algorithm to give an overall FMR of 0.0003. The 1:N identification tables compute FNIR "at the threshold that limits the False Positive Identification Rate (FPIR) to 0.003", set "for each algorithm and each column separately", with each person enrolled from one image only; the investigation tables set the threshold to zero and return a fixed 50 candidates. Two FNMR figures at different FMRs are not comparable, and a figure quoted with no FMR is not a result.

Time can be part of the protocol too. NIST requires that "Template generation duration must be under 2,000 milliseconds per image when running on a single core" from 30 November 2026. The WebFace260M paper's FRUITS protocol went further and made the budget the test: the whole system, detection to matching, must run within 100, 500 or 1,000 milliseconds on one core of a named CPU, with FNMR at a fixed FMR as the metric (arXiv 2103.04098). The WebFace260M paper explained page sets out that protocol and its test set of 2,225 identities and 38,578 faces.

04How is a result reported?

A complete report states the dataset, the protocol, the metric with its operating point, and the date. NIST's reports do all four: the 1:1 page carries participation counts (1,501 algorithms from 461 developers since 2017, and 713 algorithms from 224 developers for 1:N since 2018), a last-updated date, and tables per dataset. The pages also set the cadence: developers may submit "no more frequently than four calendar months", so a listed figure can be up to a third of a year behind a developer's current model. The NIST FRTE page follows the reports in detail.

Academic papers report on public sets with their own operating points. The ArcFace paper gives 1:1 verification on IJB-C as TAR at FAR 0.0001, reaching 0.956 for its MS1MV2 ResNet100 model (arXiv 1801.07698, Table 7); the WebFace260M paper reports the same measure, with its ResNet-100 model trained on WebFace42M at 97.70 percent against 96.03 percent for MS1MV2. Those two numbers can be compared because both state TAR at FAR 0.0001 on IJB-C; an LFW accuracy or a NIST FNMR at FMR 0.000001 cannot be set against either.

05Why do scores for the same system differ between tests?

Because the test changed, not the system. The operating point moves: a system that looks strong at FMR 0.01 can look weak at FMR 0.000001, since the second threshold is far stricter. The population moves: NISTIR 8280 found false positive rates that "often vary by factors of 10 to beyond 100 times" across demographic groups at a fixed threshold, so a test with a different mix of people yields a different rate. And the images move: NIST notes that false negatives "are strongly dependent on image quality", so a visa-photo test and a border-camera test rank the same algorithms differently.

That is why a page on this site quotes a score only with its dataset, operating point and date. The same discipline applies on the deepfake side; the MesoNet page shows one detector's scores moving from 0.984 to 0.6077 as the protocol changed around it.

06What the standards fix

ISO/IEC 19795-1:2021, Biometric performance testing and reporting, Part 1: Principles and framework, is the standard the testing bodies cite; its catalogue entry states that it "specifies performance metrics for biometric systems" and "establishes general principles for testing the performance of biometrics systems in terms of error rates and throughput rates" (ISO catalogue). The text itself is paywalled, so this site cites it by title and edition and takes its requirement to report verification and identification apart from NISTIR 8280's description of it. No figure on this page comes from the standard.