Face recognition system: the components and where errors enter

This page is the component view for the face recognition section. The how face recognition works page follows a single comparison end to end; this one takes the system apart to show which stage each published error rate describes.

01What are the components of a face recognition system?

Every face recognition system, whether it unlocks a phone or searches a passport database, runs the same chain. The camera produces an image. A face detector locates each face and returns a bounding box plus a few landmark points, usually the eyes, the nose tip and the mouth corners. An alignment step uses those points to rotate, scale and crop the face so that the eyes and mouth land in the same place in every crop. An embedding network reads the crop and outputs a fixed-length vector of numbers. A comparison step measures the distance between two vectors and applies a threshold: close enough is a match.

Two tasks use the same chain and differ only at the last step. Verification, called one to one, compares a probe against a single claimed identity, as a phone or a border gate does. Identification, called one to many, compares the probe against every entry in a gallery and returns the closest candidates, as a police search does. The error vocabulary differs between them, and NISTIR 8280 records that "the ISO/IEC 19795-1 performance testing and reporting standard requires different metrics to be reported for identification and verification implementations".

02Capture: what can go wrong before any software runs?

The camera decides what every later step has to work with. Resolution, focus, exposure, the angle of the head and the position of the light all change the pixels before any algorithm sees them, and so does the compression applied when an image is stored or transmitted. NIST's 1:1 evaluation page is explicit that false negatives "are strongly dependent on image quality, and therefore poor photography of a face can induce a demographic effect", giving under-exposure of dark-skinned individuals, over-exposure of fair-skinned subjects and a camera not adjusted for very tall or short people as examples.

Capture is also where the system is attacked. The SC37 working group that writes the biometric standards names the capture device, the data transmission channel and the data store as the three main points of a targeted attack, and the capture device is where a printed photograph, a replayed video or a mask is presented. Testing for that is a separate discipline, with its own datasets and error rates, covered on the face anti-spoofing page. A recognition error rate says nothing about it.

03Detection and alignment: how does the system find and normalise the face?

The detector is a trained network that scans the image for faces at every scale. RetinaFace, the detector the ArcFace pipeline uses, returns a box and five landmarks per face and reports an average precision of 91.4 percent on the hard split of the WIDER FACE test set (arXiv 1905.00641). Average precision summarises the two detection errors: faces that were missed, and boxes drawn around something that was not a face. In an operational system a missed face means no comparison happens at all. NIST folds this into its error rates rather than reporting it separately: NISTIR 8280 states that "a mate that is not reported on a candidate list is treated the same as a miss arising from face finding failure".

Alignment turns the five landmarks into a standard crop. The ArcFace paper generates "normalized face crops (112 × 112) by utilizing five facial points predicted by RetinaFace" (arXiv 1801.07698), and nearly every modern embedding network expects that geometry. If a landmark is misplaced, by a strand of hair, by glasses or by a profile pose, the crop is rotated or shifted and the embedding network sees a face it was not trained to read. The RetinaFace authors measured the downstream effect directly: with their detector, ArcFace's verification result on IJB-C improved to a true accept rate of 89.59 percent at a false accept rate of one in a million. A better detector raised the recognition score without touching the recognition network.

04Embedding: how does a face become a vector?

The embedding network is the component people mean when they say "the algorithm". ArcFace trains a ResNet50 or ResNet100 to map each 112 by 112 crop to a vector of 512 numbers so that two crops of the same person land close together and crops of different people land far apart, using an additive angular margin loss that the paper describes as having "a clear geometric interpretation". Extracting the 512-dimensional feature takes 8.9 milliseconds per face on ResNet50 and 15.4 milliseconds on ResNet100 in the paper's measurements. NIST times the same step on its own hardware and, from 30 November 2026, requires template generation to take under 2,000 milliseconds per image on a single CPU core.

The embedding is where the best-known systemic errors live. NIST's 1:1 page explains that false positives arise "when an algorithm produces similarity score distributions that are displaced for one demographic versus another", which "can occur due to under-representation of a demographic in the image dataset used for algorithm training". NISTIR 8280 measured those shifts across 189 algorithms in 2019 and found false positive rates that "often vary by factors of 10 to beyond 100 times" between demographic groups, with false negative differences usually below a factor of 3. Nothing in the camera or the detector produces that pattern; it is a property of the learned vector space.

05Comparison and threshold: where is the decision made?

The last component is arithmetic, but it is where the published numbers come from. Two vectors are compared to give a similarity score, and the score is compared with a threshold. Every threshold trades one error for the other: raise it and fewer impostors pass but more genuine users are rejected. NIST's 1:1 page defines the pair as "FNMR is the proportion of mated comparisons below a threshold set to achieve the false match rate (FMR) specified. FMR is the proportion of impostor comparisons at or above that threshold", and ranks algorithms by FNMR at an FMR of one in a million on its visa and border images. For one-to-many search the 1:N page sets the threshold so that the false positive identification rate is 0.003 and reports the false negative identification rate at that point.

A single accuracy figure for a face recognition system is therefore incomplete without the threshold and the task. The NIST evaluations page explains how the reports fix those two things so that algorithms can be compared, and the vocabulary of the four rates has its own page in this section.

06Who measures which errors?

NIST's tests take an image and a black-box algorithm and return the match outcome, so detection, alignment and embedding are measured together and the capture stage is fixed by the test images. Each image, the 1:1 page says, "will contain one face only". The reported FNMR therefore describes the algorithm on NIST's photographs at NIST's threshold, not a deployed system with its own cameras and its own cut-off.

07Why does the published error rate describe only part of the system?

Because the published rate is measured on fixed images with a fixed threshold, and a deployment changes both. A system scored on visa portraits meets a different distribution of lighting, angle and compression from a kiosk camera; the same embedding network produces different scores, and the threshold that gave one in a million false matches on the test set gives a different rate in the field. Compression alone moves scores measurably: the compression and deepfake detection page shows the same effect on a neighbouring task, where a strong detector loses eighteen points of accuracy between an uncompressed and a heavily compressed version of the same video.

The habit this site applies to every number follows from that. A face recognition error rate is quoted with its task, its threshold, its test images and its date, and a system vendor's accuracy claim that omits any of the four cannot be compared with a NIST figure. No single benchmark score predicts how a system performs on images it was not tested on.

08How this connects to deepfakes and voice

The same five components appear in the other systems this site covers. A face deepfake detector runs capture, detection, alignment and embedding identically and changes only the last step from "who is this?" to "is this real?", which is why its scores collapse under the same compression and distribution shifts. A speaker verification system is the audio copy of the chain, and the ASVspoof challenge series exists because its capture step can be fed synthetic speech in the same way a face system can be shown a screen. In every case the error rate on the page belongs to one component under one set of conditions, and the reader's first job is to find out which.