NISTIR 8280 explained: the NIST demographic effects report, section by section

This page is part of the face recognition section and reads the report in the order it was written. The demographic differentials page summarises what the findings mean; this one shows where each finding sits and how the report defines its terms.

01What is NISTIR 8280?

NISTIR 8280 is "the third in a series of reports on ongoing face recognition vendor tests (FRVT)" (NIST Interagency Report 8280). The first two covered the accuracy of one-to-one verification and one-to-many identification algorithms; this one asks whether that accuracy differs by sex, age and race or country of birth. The algorithms were the ones submitted to the ongoing evaluation by corporate laboratories and a few universities, described as prototypes rather than finished products.

The report has two summaries, nine numbered sections and 17 annexes. The executive summary runs four pages and holds every headline figure quoted from it; the technical summary that follows introduces the vocabulary the rest depends on. Sections 4 to 7 are the results, two chapters for verification and two for identification, and sections 8 and 9 recommend research toward mitigation. The annexes, "more than 1200 pages of charts", give the full results for each algorithm.

02Which images and algorithms did it use?

Four datasets, all "collected for authorized travel, immigration or law enforcement processes": domestic mugshots collected in the United States; application photographs from a global population of applicants for immigration benefits; visa photographs; and border crossing photographs of travellers entering the United States. The report notes that "The first three sets have good compliance with image capture standards. The last set does not", and that difference drives several of its findings about false negatives.

The mugshots carry race metadata; the other sets carry country of birth, and the analysis of them is restricted to "24 countries in 7 distinct global regions" with low long-distance immigration, so that country can stand in for region of origin. Together the sets supplied the 18.27 million images of 8.49 million people. One cross-country trial alone, on application photographs, "produced 195 billion imposter scores".

03How does the report define its terms?

The technical summary separates two ideas that casual reporting merges. Differential performance is "a difference in the genuine or imposter [score] distributions" between demographic groups. A differential outcome happens when those scores meet an operator's threshold: "A genuine score below threshold yields a false negative outcome, and an imposter score at or above threshold, a false positive outcome". The report measures outcomes, because "the vast majority of biometric systems are configured with a fixed threshold against which all comparisons are made".

Section 3 then defines the four rates formally, noting that ISO/IEC 19795-1 "requires different metrics to be reported for identification and verification implementations". For verification, FNMR is the proportion of genuine scores below the threshold and FMR the proportion of impostor scores above it; for identification, FPIR is the share of non-mated searches that return a candidate at or above threshold and FNIR the share of mated searches whose mate is missed. The FMR, FNMR, FPIR and FNIR page quotes these definitions in full. The report's reporting rule follows from them: "it is necessary to report both false negative and false positive rates for each demographic group at that threshold. This is rarely done".

04What did sections 4 and 5 find about verification?

Section 4, on false positive differentials, carries the headline. With the higher-quality application photographs, "false positive rates are highest in West and East African and East Asian people, and lowest in Eastern European individuals", with a factor of 100 between countries for many algorithms. The exception is named: "with a number of algorithms developed in China this effect is reversed, with low false positive rates on East Asian faces". With domestic mugshots, the highest false positives were in American Indians, with elevated rates in African American and Asian populations, and the ordering depended on sex and algorithm.

Two further findings sit in the same section. "We found false positives to be higher in women than men, and this is consistent across algorithms and datasets. This effect is smaller than that due to race." And "We found elevated false positives in the elderly and in children", largest at the extremes of age. Section 5, on false negatives, finds smaller and more algorithm-specific effects: higher false negatives for Asian and American Indian individuals in mugshots, and for people born in Africa and the Caribbean in the lower-quality border crossing images, which the report ties to image quality rather than to the faces.

05What did sections 6 and 7 find about identification?

Identification algorithms search a gallery, and the report observes that the gallery gives them "a resource for mitigation of demographic effects that purely one-to-one verification systems do not have". Differentials present in verification were "usually, but not always, present in one-to-many search algorithms". The important exception is stated in the executive summary: "some developers supplied highly accurate identification algorithms for which false positive differentials are undetectable". The effect is real for many algorithms and absent for some, which is why the report insists that a system owner "know their algorithm" rather than rely on a field-wide statement.

06Why does the report distinguish itself from Gender Shades?

Section 1 addresses the studies that most public discussion of face recognition bias cites. Those studies, the report says, showed "poor accuracy of face gender classification algorithms on black women", and "did not evaluate face recognition algorithms, yet the results have been widely cited to indict their accuracy". NISTIR 8280 was undertaken to measure the analogous effects in recognition itself, and it recommends that any report of bias state the class of algorithm tested. The Gender Shades study page covers the 2018 work on its own terms; the two measured different systems with different error rates.

07How should the report be read today?

Three habits carry over. Read false positives and false negatives separately, at the stated threshold, because the two move in opposite directions and affect different people. Read the algorithm, not the field: the annexes exist so that a buyer can find the one they are deploying. And treat the 2019 figures as a snapshot of the algorithms submitted by then; NIST's ongoing 1:1 and 1:N pages now carry demographic summary tables that are updated as new algorithms arrive, and the face recognition technology overview places those tables in the wider evaluation picture. The report's method has since been borrowed by the deepfake side of this site: when a detector such as GenConViT is scored on fakes from the field rather than from its training distribution, the question is the same one NIST asked of recognition, whether accuracy measured in the laboratory survives the population it meets in use.