Face recognition benchmarks: LFW, IJB-C, MegaFace, NIST FRTE and FRUITS

This page is the index for the face recognition section. It names each benchmark's protocol, metric and size from the benchmark's own paper or page, and leaves the definitions of the error rates to the metrics pages and the task split to what face recognition is.

01The five benchmarks side by side

Sizes are copied from each benchmark's paper or official page; LFW's counts come from the InsightFace datasets README. A benchmark listed as no longer distributed can still be used by groups that hold a copy, which is why its numbers go on appearing in papers.

02What does LFW measure, and why did it saturate?

Labeled Faces in the Wild (2007) fixed the protocol most later benchmarks copied: pairs of face images, each labelled same or different, and a single accuracy over the pairs. The InsightFace README records it as 5,749 identities, 13,233 images and 6,000 verification pairs. The MegaFace authors described its scale in 2015 as "13K photos of 5K people" and used that as their reason to build something a thousand times larger (arXiv 1512.00596).

Deep networks closed the benchmark within a few years. DeepID2 reported 99.15 percent in 2014, FaceNet 99.63 percent in 2015 and ArcFace 99.83 percent in 2019, a span of 0.68 points across the whole deep learning era, as the LFW dataset page shows with each paper. On 6,000 pairs, a difference of a tenth of a point is a handful of pairs, so LFW stopped separating systems and became a check that a model is not broken.

03What does IJB-C add?

The IARPA Janus Benchmark-C (2017) moved from pairs of images to templates of mixed media. Its readme describes "still images, frames and videos of celebrities and Internet personalities collected from the web", 3,531 subjects, and a design "to have no overlap with the popular face recognition benchmarks, such as University of Oxfords VGG-Face dataset and the CASIA WebFace dataset" (IJB-C readme, NIST). The media counts are 21,294 images, 117,542 video frames, 11,779 videos and 10,040 non-face images.

The benchmark defines eight tests. Test 1, 1:1 verification, has 23,124 templates and compares 19,557 genuine against 15,638,932 impostor pairs, which is why its results are quoted as a true accept rate at a false accept rate as low as one in a million rather than as an accuracy. Test 4 is a 1:N mixed search. The WebFace260M paper reports 97.70 percent TAR at FAR 10^-4 on IJB-C for a ResNet-100 trained on WebFace42M (arXiv 2103.04098), and ArcFace reported 95.6 percent at the same operating point in 2019. NIST's face challenges page states that it "discontinued distribution of all three IJB challenges (IJB-A, IJB-B, and IJB-C) on March 14th 2023".

04What did MegaFace test?

MegaFace (2015) kept the probe and gallery small and made the gallery noisy. Its paper describes "One Million photos that capture more than 690K different individuals" used as distractors, with the challenge evaluating algorithms "with increasing numbers of distractors (going from 10 to 1M) in the gallery set" and reporting both identification and verification (arXiv 1512.00596). Probes and gallery came from FaceScrub (celebrities) and FG-NET (age-invariant). The project page gives the distractor set as 1 million photos of 690,572 users and the Challenge 2 training set as 4.7 million photos of 672,057 identities.

Algorithms indistinguishable on LFW separated widely at a million distractors. The benchmark then saturated in its turn: the project page records that the challenge "concluded, reaching a benchmark performance of over 99%", and that "MegaFace is being decommissioned and MegaFace data are no longer being distributed". The faces in its distractor set were drawn from public photo collections, the same collection method that regulators later judged in the Clearview AI rulings.

05How does NIST FRTE differ from the academic sets?

NIST's Face Recognition Technology Evaluation is not a downloadable dataset. Developers submit algorithms; NIST runs them on images it holds and publishes the error rates. The 1:1 track has been open since February 2017 and has evaluated 1,501 algorithms from 461 developers, with each developer limited to submissions "no more frequently than four calendar months"; the 1:N track has evaluated 713 algorithms from 224 developers since 2018 (NIST FRTE 1:1; NIST FRTE 1:N).

The metrics are error rates at a fixed operating point, never an accuracy. For 1:1, "FNMR is the proportion of mated comparisons below a threshold set to achieve the false match rate (FMR) specified", and the leaderboard orders algorithms by FNMR at FMR 10^-6 on the visa and border set. For 1:N, FNIR is computed at the threshold that holds FPIR to 0.003, with one image per enrolled person. The NIST FRTE page explains how the reports are read and what the demographic tables add.

06What is FRUITS, and where is the WebFace260M test set?

FRUITS, Face Recognition Under Inference Time conStraint, is the protocol of the WebFace260M benchmark (2021) that this domain once hosted. Its authors argued that public evaluations were "most saturated" and that real systems are bound by inference time, so FRUITS times the entire pipeline: "The whole face recognition system must distinguish image pairs within 100 milliseconds, including preprocessing (e.g. face detection and alignment), feature embedding for recognition, and matching", measured on "a single core of an Intel Xeon CPU E5-2630-v4@2.20GHz processor" (arXiv 2103.04098). FRUITS-500 relaxes the budget to 500 ms for local surveillance systems, and FRUITS-1000, "Following NIST-FRVT", allows 1,000 ms for cloud systems. The metric is FNMR at a fixed FMR; the best lightweight baseline, a RegNet-800MF, reached 5.88 percent FNMR at FMR 10^-5 on all pairs.

The training side is WebFace42M, 2 million identities and 42 million faces cleaned from the 260 million collected, which the paper says cut the relative failure rate on IJB-C by 40 percent and placed third of 430 entries on NIST FRVT at the time. The test set used in the ICCV 2021 Masked Face Recognition challenge held 2,478 celebrities and 60,926 faces; 69 teams submitted 833 solutions in its first phase and 49 beat the organisers' baseline (arXiv 2108.07189). The challenge moved to Codalab in December 2021.

07Which face recognition benchmarks are still distributed?

Two of the five are closed. MegaFace's project page states that its data "are no longer being distributed", and NIST stopped distributing IJB-C on 14 March 2023, so a new entrant cannot obtain either from its source. LFW remains available through mirrors such as the InsightFace validation sets. NIST FRTE is ongoing and is the only one of the five whose test images were never public, which is also why its numbers cannot be reproduced outside NIST. The WebFace260M track moved to Codalab and its training set is released under a licence agreement.

A score on a withdrawn benchmark is a historical fact about one model on one date that a reader cannot re-run; the paper's own table is the only record. The deepfake side of the same problem is on the consent in deepfake datasets page.