How does face recognition work? Detect, align, embed and compare

This page opens the face recognition section. The face verification metrics page defines the error rates the last step produces, and the bridge page follows the same pipeline into deepfake detection.

01Step 1: how does the system find the face?

A face detector scans an image and returns, for each face, a bounding box and a small set of landmarks. RetinaFace, the detector used in the ArcFace line of work, is a single-stage network that predicts the box, five landmarks (the eye centres, the nose tip and the two mouth corners) and a dense 3D face shape in one pass (arXiv 1905.00641). Its authors annotated those five landmarks by hand on the WIDER FACE dataset and report an average precision of 91.4% on the WIDER FACE hard test set, 1.1 points above the previous state of the art.

Detection sets the ceiling for everything after it. A face that is missed is never compared, and a landmark that is a few pixels off shifts the crop that the embedding network sees. The RetinaFace paper measures this directly: feeding its detections to ArcFace raised verification on IJB-C to a true accept rate of 89.59% at a false accept rate of one in a million. With a light backbone the detector runs in real time on a single CPU core at VGA resolution, which is why the same detector appears in deployed systems and in research pipelines.

02Step 2: why is the face aligned before embedding?

Alignment removes the variation the network should not have to learn. Using the five landmarks, the face is rotated, scaled and cropped so that the eyes, nose and mouth land on fixed pixel positions in a crop of a fixed size. ArcFace's experimental setting states it: the authors "generate the normalised face crops (112 × 112) by utilising five facial points" (arXiv 1801.07698). Every training image and every test image passes through the same warp, so two photographs of one person differ by identity, expression, age and lighting but not by where the face sits in the frame.

Because alignment is a fixed geometric step with no learned parameters, benchmark results depend on it silently. Two systems with the same embedding network but different landmark detectors will produce different crops and different scores, which is one reason the WebFace260M paper proposes a protocol that scores the complete pipeline, detection included, under one time budget.

03Step 3: what is a face embedding?

An embedding is a vector of numbers that stands for a face so that two images of one person give nearby vectors and images of different people give distant ones. ArcFace uses a 512-dimensional embedding from a ResNet50 or ResNet100 backbone. The training trick is in the loss. Softmax classification over the training identities is replaced by a loss that first normalises both the embedding and each identity's weight vector onto a unit hypersphere, then adds an angular margin penalty of m to the angle between a face and its own identity's centre before scaling by s. The paper sets s to 64 and m to 0.5, trains at a batch size of 512, and explains that the margin corresponds exactly to a geodesic distance on the hypersphere, which gives ArcFace its name.

The result is an embedding space in which the distance between two faces is meaningful on its own, without the classifier that trained it. Trained on MS1MV2 (85,000 identities, 5.8 million images) with a ResNet100, ArcFace reports 99.83% verification accuracy on LFW and 98.02% on YouTube Faces, and a true accept rate of 94.2% on IJB-B and 95.6% on IJB-C at a false accept rate of one in 10,000 (Tables 4 and 7 of the paper). LFW itself holds 13,233 images of 5,749 people, and at 99.83% it is close to its ceiling, which is one reason current results are quoted on IJB-C and larger sets.

Figures from arXiv 1905.00641, 1801.07698 and 2103.04098, read 8 and 9 November 2026.

04Step 4: how are two faces compared?

The final step is arithmetic. The system computes the cosine or Euclidean distance between two embeddings and compares it with a threshold. Verification, the one to one case, asks whether two images show the same person; the pair is a match if the distance is under the threshold. Identification, the one to many case, compares a probe embedding with every embedding in a gallery and returns the closest entries above a threshold.

The threshold is where the error rates live. Set it loose and more genuine pairs pass but so do more impostors; set it tight and the reverse. NIST's one to one evaluation therefore fixes the false match rate first and reports the false non-match rate at that setting: "FNMR is the proportion of mated comparisons below a threshold set to achieve the false match rate (FMR) specified" (NIST FRTE 1:1). In its one to many evaluation the threshold is set so that the false positive identification rate is 0.003, and the false negative identification rate is read off at that point. The NIST evaluations page explains how those reports are published and updated.

05What decides how accurate the embedding is?

Data, mostly. The embedding network only learns identity from identities it has seen, so the number of people in the training set matters more than the depth of the network. WebFace260M (CVPR 2021) collected 4 million identities and 260 million faces from the web, cleaned them with a self-training pipeline to 2 million identities and 42 million faces, and reports that training on the cleaned set cut the relative failure rate on IJB-C by 40% and ranked third among 430 entries on NIST's test at the time (arXiv 2103.04098). Even a tenth of the data, WebFace4M, beat the public training sets then in use. The same paper proposes FRUITS, a protocol that scores a complete pipeline under a time budget of 100, 500 or 1,000 milliseconds, because detection, alignment and embedding all cost time in deployment. The WebFace260M dataset page records that history and this domain's part in it.

Accuracy is also not one number. The Gender Shades study measured a different task, gender classification, but its finding that an aggregate error rate hid a 34.4 point gap between subgroups applies with equal force to recognition, and NIST's 2019 demographic report found false match rates varying by factors of 10 to beyond 100 across groups in many algorithms.

06Where does this pipeline show up outside recognition?

Everywhere a system needs to know whose face it is looking at. Face swap generators such as SimSwap use a pretrained ArcFace model to extract the identity they inject into a target video, and the face swap deepfakes page shows swap quality being scored with a recognition network. Deepfake detectors run the same detect and align steps before their own classifier; the Xception detector that every detection benchmark uses as its baseline is an ImageNet classifier fed aligned face crops. Liveness and presentation attack detection run them too. The recognition pipeline is the common front end, and its measurement habits, fixed protocols, thresholds set by false match rate and demographic breakdowns, are the habits the rest of this site holds detection to.