MegaFace: the million-distractor benchmark and its retirement

This page is part of the face recognition section and follows the LFW dataset page, because MegaFace exists as an answer to LFW: its authors wrote that "LFW includes only 13K photos of 5K people" and that evaluation needed to move to the million scale.

01What did MegaFace measure?

MegaFace tested identification and verification with a gallery that grew from 10 to 1 million distractors, faces of people who are not the person being searched for. A probe photo of a known identity was matched against the gallery plus the other photos of that identity, and the question was whether the correct person came out at rank 1 as the number of strangers in the gallery rose by five orders of magnitude. The paper presents both identification and verification, evaluates "with respect to pose and a person's age", and compares results "as a function of training data size" (Kemelmacher-Shlizerman, Seitz, Miller, Brossard, 2015).

Two probe sets supplied the known identities. FaceScrub provided celebrities; the evaluation used a subset of 80 identities, 40 women and 40 men, with 50 photos each. FG-NET provided 975 photos of 82 people with large age differences, more than 40 years for some. The gallery of distractors came from Yahoo's 100 million image Flickr set, released under Creative Commons licences, sampled across about 500,000 Flickr user IDs by taking the first photo from each user with a face larger than 50 by 50 pixels. The project page gives the final counts: 1 million photos and 690,572 unique users.

02How far did LFW-level algorithms fall at the million scale?

The paper's first stated finding is the one worth quoting: "Algorithms that achieve above 95% performance on LFW (equivalent of 10 distractors in our plots), achieve 35-75% identification rates with 1M distractors." Two classical baselines, joint Bayesian and LBP, dropped "to less than 10%". The challenge was released on 30 September 2015 with three weeks for evaluation; more than 100 groups registered and the paper reports results from the five that uploaded complete features. In the paper's Figure 6, rank-1 identification with 1 million distractors on the FaceScrub probe set ranged from 33.70 percent to 73.30 percent across the seven entries, with FaceNet, which the paper says was trained on more than 500 million photos, at 70.49 percent, and 74.59 percent on the FG-NET probe set.

The paper also explains why the LFW verification protocol could not show this. Results on LFW "are typically reported at equal error rate which implies false accept rate of 1%-5% for top algorithms", whereas for a gallery the size of MegaFace "only FAR of 10^-5 or 10^-6 is" meaningful. A false accept rate acceptable on 6,000 pairs produces thousands of false matches against a million strangers. The verification metrics page in this section defines the rates involved; this is the benchmark that made the low-FAR operating point the standard way to quote them.

03What was the MF2 training set?

The first challenge let groups train on any data, and the paper observed that algorithms "trained on larger sets" generally did better. The follow-up, MF2, fixed that variable. Its authors describe "a benchmark, MF2, that requires all algorithms to be trained on same data, and tested at the million scale", built on "a public large-scale set with 672K identities and 4.7M photos" (Nech and Kemelmacher-Shlizerman, 2017). The project page lists the training set as 4.7 million photos of 672,057 unique identities, with a mean of 7 photos per person and a range of 3 to 2,469.

MF2's reported findings were that algorithms trained on it reached results comparable with those trained on massive private sets, that some "outperformed themselves once trained on MF2", and that "invariance to aging suffers from low accuracies as in MegaFace". The last point matched the first challenge, where FG-NET, the age-varied probe set, was harder than FaceScrub with 10 distractors.

04Why was MegaFace decommissioned?

The project site gives two reasons in one sentence: the challenge "has concluded, reaching a benchmark performance of over 99%", and "ongoing maintenance of this platform would require considerable administrative effort". The consequence is stated plainly: "MegaFace is being decommissioned and MegaFace data are no longer being distributed" (megaface.cs.washington.edu). The notice names no date and no complaint; it describes a benchmark that had done its job. A 2015 measure built to separate algorithms clustered above 95 percent on LFW had, by its own leaderboard, been pushed past 99 percent.

The retirement sits inside a wider pattern in face recognition data. MS-Celeb-1M, a 10 million image training set of celebrities, is no longer served by its original site; NIST discontinued distribution of the IJB-A, IJB-B and IJB-C challenge sets on 14 March 2023. The face datasets and consent page gathers those withdrawals with their stated reasons, which differ: MegaFace's notice cites completed goals and maintenance cost, not consent, even though its gallery was built from Creative Commons Flickr photographs drawn from a collection that, in the paper's words, "was not created with face analysis in mind".

05What does the MegaFace story teach about benchmarks?

That a benchmark measures the gap between the data and the algorithms of its moment, and closes when the gap does. In 2015 the move from 10 distractors to a million cut rank-1 identification from above 95 percent to a 35 to 75 percent band for the same algorithms; by the time of the retirement notice the leaderboard had passed 99 percent. The same arc is visible in deepfake detection a decade later, where detectors above 0.93 frame-level AUC on their training benchmark fall to 0.55 to 0.95 on datasets they never saw, as the frame-level versus video-level AUC page and the 25 paper reading list record. The lesson MegaFace taught recognition, that a score needs its gallery size, its false accept rate and its date attached, is the one this site applies to every detection number.

Regulation has started to encode the same caution in its own terms. The EU AI Act Article 50 page describes a transparency duty framed around marking content rather than around any benchmark accuracy, and face recognition systems have their own provisions elsewhere in that regulation. A million-distractor result from 2015 and a 2024 regulation agree on one point: a headline percentage without its conditions says little.