IJB-C: 3,531 subjects, and the face verification test NIST stopped distributing

This page belongs to the face recognition section. It explains the benchmark behind many scores on the list of face recognition benchmarks, and the format those scores use.

01What is in IJB-C?

IJB-C is the third release of the IARPA Janus Benchmark series. The NIST README describes its contents as "still images, frames and videos of celebrities and Internet personalities collected from the web", and states that "There are 3531 subjects in the Database". The same document says the set is "designed to have no overlap with the popular face recognition benchmarks", and names the VGG-Face and CASIA WebFace datasets as examples.

The media counts in Table 1 of the README are 21,294 images, 117,542 frames, 11,779 videos and 10,040 non-face images. The non-face images exist for the detection test, which asks a system to find faces and to reject pictures that contain none. The README also documents per-image metadata, including face occlusion, so a result can be broken down by condition rather than only reported as one number.

02How is IJB-C scored?

The README defines eight tests. Test 1 is 1:1 verification, Test 2 is 1:1 covariate verification, Test 4 is 1:N mixed search, Test 6 is face detection, Test 7 is clustering, and Tests 9, 10 and 11 are wild probes from still images and frames, from video and from mixed media. The scores quoted in recognition papers almost always come from Test 1.

Test 1 builds templates, each a set of images and frames of one subject, and compares them in pairs. The README gives its size in Table 3: 3,531 subjects, 23,124 templates, 19,557 genuine matches and 15,638,932 impostor matches. A genuine match compares two templates of the same person, an impostor match two different people. Test 2 is larger, with 140,739 templates and 39,584,639 impostor matches, and compares single still images so that failures can be traced to a cause such as yaw difference. Test 4 is an open-set search with 19,593 templates.

The verification result is reported as the true accept rate at a chosen false accept rate, described on the TAR at FAR page. The false accept rate is the share of impostor matches wrongly accepted, so a quoted figure means little without it.

03What do published IJB-C scores look like?

The same model gives different numbers at different false accept rates, which is why the operating point has to travel with the score. The table lists scores that appear in papers already used on this site, each with its protocol as the paper states it.

A stricter false accept rate lowers the true accept rate of any model, so the last row cannot be ranked against the others. The first five rows share one operating point and show the gain from model and training set: the same ArcFace loss on a ResNet-100 rises from 96.03 to 97.70 percent when the training set changes from MS1MV2 to WebFace42M. ArcFace and the WebFace260M paper both report at 1e-4.

04Why do the counts differ between sources?

Three documents give three sets of numbers for the same database. The NIST face challenges page says IJB-C "defines eight challenges" on "138000 face images, 11000 face videos, and 10000 non-face images". The README gives 21,294 images and 117,542 frames. The ArcFace paper describes it as "3, 531 subjects with 31.3K still images and 117.5K frames from 11, 779 videos".

They agree once the arithmetic is done. Adding images and frames gives 138,836, in line with NIST's rounded 138,000 face images. Adding the 10,040 non-face images to the 21,294 images gives 31,334, in line with ArcFace's 31.3K stills. The ArcFace paper also lists IJB-C as 148.8K images or videos in its Table 1, which is its 31.3K stills plus 117.5K frames. When a paper cites a size for IJB-C, check which of these groupings it means.

05What did NIST withdraw in 2023?

The NIST face challenges page states that "NIST has discontinued distribution of all three IJB challenges (IJB-A, IJB-B, and IJB-C) on March 14th 2023". The page gives no reason, and this page does not speculate about one. The practical effect is that new work cannot obtain the data from NIST. Copies circulate elsewhere: the InsightFace datasets README lists IJB-B and IJB-C as image test suites with the data on Google Drive.

The licence is part of the data. The README says the database is "made available under different Creative Commons license variants found in the license/ directory", requires users to comply with the terms in license/*.csv, and warns that it is "subject to change at any time". The terms attach to each image, so a copy is only as usable as its licence file.

06Is IJB-C still a useful benchmark?

It remains the reference for the score format, and the figure for a method on IJB-C at 1e-4 is a standard line in recognition papers. Three of the 1e-4 rows above sit between 96.03 and 97.7 percent for ResNet-100 models, a narrow band. NIST's FRTE evaluation ranks algorithms by FNMR at a fixed FMR on its own operational image sets, which is a different test for a different question. A score on IJB-C is a score on 3,531 celebrity subjects and a fixed protocol, not a prediction for a border or an airport.