Face embeddings: what the vector is and how similarity is scored

This page is part of the face recognition section. It expands step three of the four-step pipeline and feeds the verification metrics that every benchmark table on this site reports.

01What is a face embedding?

An embedding is the output of the last layer of a recognition network, a list of numbers of fixed length that stands in for the face. FaceNet describes it as "a mapping from face images to a compact Euclidean space where distances directly correspond to a measure of face similarity", produced by "a deep convolutional network trained to directly optimize the embedding itself" (arXiv 1503.03832). The vector has no named parts; no element means nose width or eye colour. Its only property is that faces of one person land close together and faces of different people land far apart.

The length is a design choice. FaceNet uses 128 floating-point values during training and shows they can be stored as 128 bytes without loss of accuracy. ArcFace extracts "the 512-D features" from a ResNet50 or ResNet100 in 8.9 or 15.4 milliseconds per face (arXiv 1801.07698). In both cases the input is a crop that has already been detected and aligned; ArcFace uses 112 by 112 pixel crops positioned by five landmarks from RetinaFace. A template for a video or a set of stills is the mean of its frame embeddings, which is how ArcFace scores IJB-B and IJB-C.

02How is the embedding trained?

FaceNet trains with a triplet loss: for an anchor face, a second image of the same person and an image of someone else, the loss pushes the anchor closer to the match than to the non-match by a margin, with the hard triplets chosen online during training. The network that set the LFW record of 99.63 percent was trained on 100 million to 200 million face thumbnails of about 8 million identities, a private set, and the paper reports that tens of millions of images cut the error by 60 percent relative to millions.

ArcFace trains as a classifier over the identities in its training set and changes the final layer so that the classifier's score is a cosine. The feature and each class weight are L2-normalised, so "the predictions only depend on the angle between the feature and the weight", and the embeddings "are thus distributed on a hypersphere with a radius of s". An additive angular margin m is then added to the angle to the true class before the softmax, with s set to 64 and m to 0.5. Because the loss is written in angles, the embedding it learns is meant to be compared by angle, which is why cosine similarity is the standard comparison for ArcFace-style models.

03How many dimensions does an embedding need?

Fewer than most deployments use. FaceNet's Table 5 holds the network fixed and varies the embedding size, reporting the validation rate at a false accept rate of one in a thousand on its hold-out set: 86.8 percent at 64 dimensions, 87.9 at 128, 87.7 at 256 and 85.6 at 512, with the differences described as statistically insignificant. The paper chose 128 and notes that smaller embeddings "are possible at a minor loss of accuracy".

ArcFace's 512 dimensions therefore reflect convention and the capacity of a larger training set rather than a measured need. The practical consequence is storage and search cost, which grow with the width of the vector and the size of the gallery, and which FaceNet cites as the reason its 128-byte vector "is ideal for large scale clustering and recognition". The accuracy a model reaches depends far more on its training data and loss than on the width of the vector, as the FaceNet training-size experiment shows.

04How is similarity scored between two embeddings?

Verification compares two embeddings and returns one number. FaceNet uses the squared L2 distance: a distance of 0.0 means identical embeddings and, in the paper's illustration, 4.0 is the opposite end, two different identities. The paper states that "face verification simply involves thresholding the distance between the two embeddings; recognition becomes a k-NN classification problem". A pair whose distance falls at or below the threshold d is called the same; the set of same-identity pairs accepted at d gives the true accept rate and the set of different-identity pairs accepted at d gives the false accept rate.

ArcFace compares normalised vectors, so the score is the cosine of the angle between them, 1.0 for identical directions and lower as the angle grows. The decision is the same shape: a cosine at or above a threshold is a match. Both scores sit on a continuous scale, and the benchmark result is a curve over every threshold rather than one accuracy; the face verification metrics page explains how that curve is summarised and why the operating point must be stated with the number.

05Where does the threshold come from?

From the data and the error the deployment is willing to accept. On LFW, FaceNet selected the distance threshold on nine training splits and applied it to the tenth; the chosen value was 1.242 for every split but one, where it was 1.256. That protocol fixes the threshold before the test pairs are seen, which is what makes the 99.63 percent accuracy a measurement rather than a fit.

Operational evaluations set the threshold from the false match side. NIST's FRTE 1:1 reports FNMR as "the proportion of mated comparisons below a threshold set to achieve the false match rate (FMR) specified", so each algorithm's threshold is whatever produces the stated FMR on NIST's images, and the NIST FRTE reports quote the miss rate at that point. NIST SP 800-63A asks remote identity proofing to run 1:1 comparison at a false match rate of 1 in 10,000 or better, so the threshold is chosen for that rate and the resulting false non-match rate, required to be 1 in 100 or better, is the cost. Change the threshold and both numbers move in opposite directions; a benchmark score without its threshold rule is not comparable to another.

06What does the embedding not tell you?

Three things that a verification flow still has to establish. It does not say whether the face was live: an embedding computed from a printed photograph or a replayed screen matches the person on it, which is why presentation attack detection is tested as a separate stage, as the spoofing and deepfake tests page sets out. It does not say whether the face was manipulated: a swap that preserves the target's identity produces an embedding close to the target's, and face swap generators such as SimSwap use an ArcFace identity vector as the signal they preserve; joint face and voice detectors learn a separate audio-visual representation for that question. And it does not reveal its training set: ArcFace's headline models were trained on MS1MV2 and MS1MV3, cleaned derivatives of a set that was taken down in 2019, and nothing in a 512-value vector shows that.

07Why numbers differ between sources

FaceNet reports 99.63 percent on LFW and ArcFace reports 99.83 for its MS1MV2 ResNet100 model, and the two are not a like-for-like comparison of embeddings. The training sets differ by two orders of magnitude in identities, the losses differ, the crops differ, and ArcFace's own paper prints 99.83 in one table and 99.82 in another for the same model. A distance threshold of 1.242 is also not convertible to a cosine threshold without knowing the scale of the vectors. When two papers quote a verification figure, the comparison that holds is the one made on the same test set under the same protocol, at a stated false accept rate.