NIST FRVT reports explained: how to read one algorithm's report card

This page belongs to the face recognition section. The NIST FRTE explainer describes the evaluation as a whole; this page is about reading one of its outputs without over-claiming.

01What is a report card?

NIST's ongoing face recognition evaluation, started in February 2017 as FRVT and "split and renamed to FRTE and FATE" on 18 August 2023, publishes its results in two forms: summary tables on the track pages, and a report card per algorithm (NIST FRTE 1:1). The header of the one read for this page gives the format: developer name, algorithm name, "Algorithm Type: 1:1 Verification", "Date of Algorithm Submission: 2017_03_16" and "Date of Report Card Generation: 2024-03-27".

Two features of that header matter before any plot is read. The algorithm is a specific submission, identified by a number after the developer's name, not a product; developers may resubmit "no more frequently than four calendar months", and each submission gets its own card. And the card date is separate from the submission date: a card can be regenerated years after the algorithm arrived, when NIST adds a dataset or an analysis, so the plots describe that submission under the evaluation as it stood on the card date.

02What does the header tell you?

The 1:1 track page draws one row per developer in its headline table, choosing "the one with the lowest mean of within-developer FNMR ranking where the mean is estimated over six columns", with ties broken by submission date. The card for any other submission from the same developer is still published, and the first section of every card, "Developer gains", plots that developer's accuracy over time across all its submissions.

The participation statistics give the scale. Since 2017 the 1:1 track has received 1,501 algorithms from 461 unique developers, with between 57 and 224 submissions in a year; the 1:N track has received 713 algorithms from 224 developers since 2018 (NIST FRTE 1:N). A card therefore describes one of more than two thousand submissions, and a claim that an algorithm "was tested by NIST" is true of all of them.

03Which datasets are on the card, and why does it matter?

Every panel names the image set it was computed on, and the sets are not interchangeable. The 1:1 page orders its table by rank on the VISABORDER dataset, visa portraits compared with border crossing photos, and the card's comparative accuracy section draws a separate panel per dataset showing "false non-match rates vs. false match rates" for the target algorithm beside "several of the most accurate algorithms listed in the caption", which change from dataset to dataset. The demographic panels compare border crossing photos against application portraits; the ageing panels use mugshots.

The 1:N page spells out why the header rows exist: "the first row in the header shows the type of image enrolled in the gallery; the second row shows the search image type; the third row shows the number of persons in the gallery", and "in all cases, each person is enrolled with one image only". An identification error rate without its gallery size is not a result. The NISTIR 8280 report page describes the four operational datasets behind the demographic analyses.

04What operating point was the threshold set to?

Each error rate on a card is read at a threshold fixed to produce a stated rate of the opposite error. The 1:1 definition is the one to memorise: "FNMR is the proportion of mated comparisons below a threshold set to achieve the false match rate (FMR) specified. FMR is the proportion of impostor comparisons at or above that threshold." The headline ranking uses FNMR at FMR 10⁻⁶; the demographic FMR summary fixes the threshold "to give FMR if 0.0003 overall"; the demographic FNMR summary and the ageing panels use a threshold giving FMR 0.00001; the twins table uses FMR 0.0001 on mugshots. The card's country-pair heatmaps fix a global threshold giving FMR 0.0001 and warn that "each +1 increase in log10 FMR corresponds to a factor of 10 increase in FMR".

For 1:N cards the fixed quantity is the false positive identification rate: "The FNIR values are computed at the threshold that limits the False Positive Identification Rate (FPIR) to 0.003", and "the threshold is set for each algorithm and each column separately". A second table reports the investigation mode, where "the threshold is set to zero and the algorithm returns a fixed number (50) of candidates" and the error is a mated search whose correct candidate is not at rank 1. The four error rates page defines each term; the face verification metrics page translates them into the TAR at FAR convention that papers use.

05How are the demographic panels read?

A 1:1 card has two demographic blocks, and the track page explains why they are read differently. False negatives "are strongly dependent on image quality, and therefore poor photography of a face can induce a demographic effect"; false positives "arise when an algorithm produces similarity score distributions that are displaced for one demographic versus another", which "can occur due to under-representation of a demographic in the image dataset used for algorithm training". The false negative panels plot FNMR by country of birth and by age above or below 45, with "bootstrap 95-percent confidence intervals around the point estimate" and the instruction that "overlapping intervals is an indication of no significant difference". The false positive panels fix the threshold to give FMR 0.001, and in a second row 0.0001, for the group that usually has the lowest FMR; the card states that "the country of birth is used as a proxy for race".

The summary table on the track page compresses each card's demographic block into seven columns: name, submission date, a summary FNMR, the best and worst FMR with their region, sex and age group, "the maximum FMR over the geometric mean FMR", where "the ideal result is 1, indicating parity", and a Gini coefficient of how false matches concentrate, "small values are better". The demographic differentials page sets out what the 2019 report found across 189 algorithms; a single card shows where one submission sits against that pattern.

06What mistakes produce a false ranking?

The cards are built to make like-for-like comparison possible, and most misreadings come from dropping one of the conditions. Comparing an FNMR at FMR 10⁻⁶ on VISABORDER with an FNMR at FMR 0.00001 on mugshots compares two operating points and two datasets at once. Quoting a developer's best card as the product's accuracy ignores that the track page picks its row by a six-column rank, and that a 2017 card is still online beside a 2026 one. Reading a 1:N FNIR without its gallery size, or an investigation-mode rank 1 figure as if it were a thresholded result, mixes two tables that the page keeps apart. And treating the tables as stable ignores that the report "is a draft made available for comments" and "will be updated on a monthly basis", that values in blue use a 2022 API change allowing multiple faces per image, which "occurs in approximately 3% of border images and 7% of kiosk images", and that from 30 November 2026 template generation must run "under 2,000 milliseconds per image" on one core, a rule that will retire some older submissions from new tables. The DFDC Preview paper made the same case for fixed operating points in deepfake detection, where prevalence makes an unanchored precision meaningless.

One more limit sits outside the card entirely. A report card measures whether an algorithm matches faces; it does not test whether the face in front of the camera was a live person, a screen or an injected video. Presentation attack detection has its own NIST track and its own standard, and the deepfake detection and prevention layers page shows where each check belongs.