NIST's face recognition evaluations (FRVT, now FRTE): what they test and how to read them

This page is part of the face recognition section and reads NIST's own pages; it names no vendor and ranks nothing. The quantities it reports are explained on the face verification metrics page.

01What NIST evaluates

The programme now called FRTE (its pages still carry the FRVT name in their addresses) takes algorithms, not products: a developer submits a library that NIST links into its own test harness and runs on sequestered datasets. Because NIST holds the images and the developers never see them, a submission cannot be tuned to the test set, which is the property most academic benchmarks lack. The images are operational: mugshots, visa application photographs, border crossing images and similar, drawn from government holdings, rather than photographs collected from the web. That choice makes the results relevant to the settings where face recognition is deployed by governments and irrelevant, or at least untested, for photographs of the kind found on social platforms.

Results are published as web tables and per-algorithm report cards rather than as one ranking, and the 1:1 page states that it "will be updated on a monthly basis as algorithms and computations complete". Submissions from any one developer are accepted no more frequently than every four calendar months, so a table at any moment mixes algorithms of different ages (NIST FRTE 1:1).

02The tracks

The evaluation runs as separate tracks, each with its own pages. The 1:1 verification track measures whether two images of a face match. The 1:N identification track measures searches of a probe against a gallery, with the threshold "set to limit the False Positive Identification Rate (FPIR) to 0.003" in its headline tables (NIST FRTE 1:N). Other tracks exist, and a dedicated 2019 report covered demographic effects. Liveness and presentation attack detection are not part of these tracks; they are tested under ISO/IEC 30107-3 by separate laboratories, as the liveness page explains.

03How accuracy is reported

The 1:1 page defines its quantities in one sentence each: "FNMR is the proportion of mated comparisons below a threshold set to achieve the false match rate (FMR) specified. FMR is the proportion of impostor comparisons at or above that threshold." A table cell is therefore an FNMR at a stated FMR, on a stated dataset, for a stated algorithm version, and three of those four fields are column or row headings that a reader must carry along with the number. The 1:N page does the same: "FNIR is the proportion of mated searches failing to return the mate above threshold. FPIR is the proportion of non-mated searches producing one or more candidates above threshold." These are the NIST names for the true accept and false accept rates academic papers write as TAR and FAR; the verification metrics page gives the translation and why the operating point cannot be dropped.

04Reading a NIST report without over-claiming

Datasets differ from the wild

An algorithm's FNMR on visa photographs, which are frontal, well lit and compliant with a standard, says little about its behaviour on a phone photograph taken at an angle in a dark room. NIST's own tables show the same algorithm with very different error rates across its datasets, which is the point of publishing them separately. A claim that quotes one dataset's figure as "the" accuracy has dropped a field.

Vendor claims versus NIST tables

The tables are per algorithm version, dated, and tied to a dataset and operating point. A sentence of the form "ranked first by NIST" without the track, dataset, operating point and date cannot be checked and usually describes one cell of one table at one time. NIST publishes no overall ranking, and the four-month submission cadence means the cell may have been overtaken. The habit this site applies everywhere, source, test set, protocol and date beside every number, is what NIST's own pages model.

05Demographic differentials

In December 2019 NIST published NISTIR 8280, Face Recognition Vendor Test Part 3: Demographic Effects, which ran 189 mostly commercial algorithms from 99 developers over 18.27 million images of 8.49 million people and reported how false match and false non-match rates varied by sex, age and race or country of birth (NISTIR 8280). Its findings, that false positive differentials are far larger than false negative ones and vary by factors from ten to beyond a hundred across groups for many algorithms, are set out with their conditions on the demographic differentials page. How the training data behind such differences were gathered is the subject of the datasets and consent page.

06Liveness is tested separately

Nothing in the FRTE tables says whether an algorithm can tell a live person from a photograph, a screen or an injected video. The matching accuracy they report assumes a genuine capture. A deployment needs both figures, from two different kinds of test, and the liveness and injection attacks page covers the second.