What is face recognition? A definition, the two tasks and their scoring

This page is the entry point of the face recognition section. The mechanics are kept to one section here because the how face recognition works page covers detection, alignment, embedding and matching with the papers behind each step; this page is about what the term means, what it excludes, and how a result is scored.

01A definition in one paragraph

A face recognition system takes an image containing a face and produces a decision about whose face it is. It does so by converting the face into a fixed-length vector, called an embedding or template, and measuring how close that vector lies to vectors made from faces of known identity. ArcFace, the most cited modern method, produces a 512-dimensional embedding from a 112 by 112 pixel crop aligned on five facial landmarks (arXiv 1801.07698), and two images of the same person are expected to produce vectors that lie close together while images of different people lie far apart. The decision is made by comparing the distance, or a similarity score derived from it, against a threshold.

Three things follow from the definition. Face recognition always involves at least two faces, the one being examined and the one or many it is compared against. Its output is a score before it is a decision, and the threshold that turns the score into a yes or no is a choice made by the operator, not by the algorithm. And its errors come in two kinds, matching the wrong people and failing to match the right ones, which is why no single number describes it.

02What face recognition is not

Face detection finds faces in an image and returns a box and, often, a few landmarks; it says nothing about identity. RetinaFace, for instance, predicts a box and five landmarks for every face it finds (arXiv 1905.00641) and is a step inside a recognition pipeline, not a recognition system. A camera that draws rectangles around faces is detecting, not recognising.

Attribute classification labels a face with a property such as apparent age or gender, with no reference to identity. The Gender Shades study of 2018, which found error rates of up to 34.7 percent on darker-skinned women in three commercial gender classifiers, is about this task, not about matching identities. Deepfake detection asks whether a face image has been synthesised or manipulated and makes no comparison with a stored identity; it shares the detection, alignment and embedding steps but answers a different question, which the deepfake detection section covers. Liveness or presentation attack detection asks whether the face in front of the camera is a live person rather than a photo, screen or mask, and is scored by its own error rates.

03How does face recognition work, in brief?

Every modern system runs the same four steps. A detector locates the face and its landmarks. The face is aligned, meaning rotated and scaled so that the landmarks fall on fixed positions, and cropped to a standard size, 112 by 112 pixels in the ArcFace family. A trained network turns the crop into an embedding; ArcFace's training loss, an additive angular margin, pushes embeddings of one identity together and different identities apart, and the authors report 8.9 milliseconds per face for a ResNet-50 and 15.4 for a ResNet-100 (arXiv 1801.07698). The embedding is then compared with stored embeddings by a distance or cosine similarity.

Training data is what made these embeddings work. The networks are trained to tell apart tens of thousands to millions of identities collected from the web, and the size of that collection tracked the gains on every benchmark through the 2010s; the sets and their consent positions are documented across this section. The per-step detail, with the papers for each, is on the how it works page linked above.

04What are the two tasks, verification and identification?

Verification is a 1:1 comparison: one probe face against one claimed identity, as when a phone unlocks or a passport gate compares the traveller with the chip photo. The system answers whether the two embeddings belong to the same person. Identification is a 1:N search: one probe against a gallery of N enrolled identities, returning the closest candidates above a threshold, as in a police search of a mugshot database. NIST evaluates the two separately, and in its 1:N track "each person is enrolled with one image only" (NIST FRTE 1:N).

The two tasks fail differently. In verification a false match lets an impostor through and a false non-match turns away the right person. In identification a false positive returns a stranger as a candidate, and the more identities the gallery holds, the more chances there are for a stranger to score above the threshold, so a 1:N error rate cannot be read off a 1:1 one without knowing N. The 1:1 versus 1:N page works through the consequences, and NISTIR 8280 records that the ISO/IEC 19795-1 performance testing standard "requires different metrics to be reported for identification and verification implementations".

05How is face recognition scored?

Verification is scored by two error rates at a chosen threshold. In NIST's words, "FNMR is the proportion of mated comparisons below a threshold set to achieve the false match rate (FMR) specified. FMR is the proportion of impostor comparisons at or above that threshold" (NIST FRTE 1:1). NIST ranks algorithms by FNMR at FMR 10^-6, one false match in a million impostor comparisons, on its visa and border photographs. Academic papers report the same quantity the other way up, as a true accept rate at a fixed false accept rate; ArcFace, for example, reported 95.6 percent TAR at FAR 10^-4 on IJB-C in 2019.

Identification is scored by its own pair. "FNIR is the proportion of mated searches failing to return the mate above threshold. FPIR is the proportion of non-mated searches producing one or more candidates above threshold", and NIST computes FNIR "at the threshold that limits the False Positive Identification Rate (FPIR) to 0.003". A second mode, investigation, sets the threshold to zero and returns 50 candidates for a human to review, scoring the share of searches where the right person is not at rank 1. A single accuracy figure, such as the 99.83 percent that ArcFace reported on the LFW pairs, belongs to a third protocol, a balanced pair test, and cannot be converted into any of the four rates above without the operating point. No face recognition figure means anything without its task, its threshold and its test set.

06Where is face recognition evaluated?

Published numbers come from a handful of benchmarks. LFW (13,233 images, 6,000 pairs) was the standard until deep networks saturated it above 99 percent; IJB-C (3,531 subjects) became the academic reference for TAR at FAR; MegaFace tested identification with a million distractors before it was decommissioned; and NIST FRTE, which has evaluated 1,501 algorithms from 461 developers in its 1:1 track and 713 from 224 in its 1:N track, is where commercial systems are compared on images that were never public. The list of face recognition benchmarks page tabulates all five with protocol, metric and status.

Two further matters sit beside the scores. Error rates differ by demographic group: NISTIR 8280 found false positive rates that "often vary by factors of 10 to beyond 100 times" across demographics for many algorithms, which is why a benchmark result should be read with its demographic tables. And the faces in the training and test sets are personal data: the Clearview AI rulings record what five regulators decided about a database of scraped faces, and the consent in deepfake datasets page carries the same question into the detection sets.

Police and government use is the most audited application; the audits of law enforcement face recognition page lists the findings by report and date.