AI benchmarks explained: what a benchmark is and how it saturates

This page is the bridge between this site's two fields. It belongs to the deepfake detection hub and uses face recognition as its first example because that field saturated a decade earlier; the LFW dataset page carries the recognition story in full.

01What is an AI benchmark?

A benchmark is a dataset, a protocol and a metric, and all three have to be named before a score means anything. The dataset is the fixed collection of items: 13,233 face images of 5,749 people in LFW, as the InsightFace datasets README records it, or 1,000 source videos and 4,000 manipulated ones in FaceForensics++ (arXiv 1901.08971). The protocol says what a system is shown and what it must answer: 6,000 pairs of LFW images with a same-or-different answer for each, or a frame from a video with a real-or-fake answer. The metric reduces the answers to one figure: verification accuracy over the 6,000 pairs, or the area under the ROC curve over all frames.

A score quoted without one of the three is not a benchmark result. "Accuracy of 99.63 percent" is meaningless until it is "LFW verification accuracy of 99.63 percent over the 6,000 pair protocol", which is how the FaceNet paper reports it (arXiv 1503.03832). The same discipline applies to any AI benchmark, from language model leaderboards to medical imaging tests, and the two fields on this site are small enough to show every part of the mechanism with real tables.

02Why does a benchmark saturate?

Saturation is what happens when the metric stops separating systems. A fixed dataset has a fixed amount of difficulty in it; once a class of methods can solve almost every item, the remaining errors are label noise, genuinely ambiguous cases, or a handful of hard items that every entrant eventually fits. The number climbs toward its ceiling, the gaps between entrants shrink below the noise of the test, and new systems can no longer be ranked by it.

LFW is the clearest case this site covers. The CASIA-WebFace paper noted in 2014 that groups with private training data were already at 97 to 99 percent (arXiv 1411.7923). DeepID2 reported 99.15 percent the same year (arXiv 1406.4773), FaceNet 99.63 percent in 2015, cutting the previous error by 30 percent, and ArcFace 99.83 percent in 2019 (arXiv 1801.07698). On 6,000 pairs, the difference between 99.63 and 99.83 is twelve pairs. A benchmark whose entire top tier fits inside twelve decisions is saturated, and the field moved its attention to harder protocols: IJB-C at a fixed false accept rate, MegaFace with a million distractors, and the NIST FRTE tests, which is also the evaluation that deployments such as TSA's checkpoint face comparison point to when they describe their algorithms.

03Did face recognition datasets saturate the same way?

They did, one after another, and each time the response was a larger or harder set rather than a better metric. MegaFace added one million distractor photos of 690,572 people to the gallery so that identification had to work against a crowd; its project page now states that the challenge "has concluded, reaching a benchmark performance of over 99%" and that the data are no longer distributed (megaface.cs.washington.edu). WebFace260M went the other way and scaled the training side: 260 million faces of 4 million identities, cleaned to 42 million faces of 2 million identities, with a time-constrained evaluation protocol called FRUITS because accuracy alone no longer separated systems (arXiv 2103.04098).

The pattern is the same in each case. A benchmark is useful while the metric still spreads the entrants out. Once the leaders bunch at the ceiling, the community either retires the set, as MegaFace was, or changes the protocol so that the question is harder, as FRUITS did by adding an inference time budget.

04What does saturation look like in deepfake detection?

DeepfakeBench Table 3 shows it in one row of numbers. Fifteen detectors were trained on FaceForensics++ at compression level c23 and tested on the same set's test split. Ten of the fifteen score a within-domain average AUC above 0.93, and the FF-c23 column tops out at UCF 0.9705, CORE 0.9638, Xception 0.9637 and F3Net 0.9635 (arXiv 2307.01426). The paper's authors describe the result plainly: plain CNN classifiers "register high AUC values that are comparable to more sophisticated algorithms". The cross-dataset results grid on this site prints the whole table.

The deepfake version of saturation has a twist that face recognition did not. When the same fifteen detectors are tested on sets they were not trained on, the averages fall to between 0.6551 and 0.7875, and the ranking changes: SPSL, seventh in the FF-c23 column, leads the cross-domain average at 0.7875. The in-dataset number saturated while the out-of-dataset number did not, which is why this site ranks detectors by the cross-domain column and not the within-domain one. The in-the-wild benchmarks go further: on Deepfake-Eval-2024, the best open-source video detector off the shelf scored 0.63 AUC against 0.96 on the benchmark it was built for (arXiv 2503.02857).

05How should any AI benchmark number be read?

Four fields turn a number into a claim. The first is the dataset and split: which items, which version, and whether the tested items overlap the training items. The second is the protocol: pairs or single items, frame level or video level, which compression, and whether the test is within the training distribution or outside it. The third is the metric: accuracy at a threshold, AUC over all thresholds, or an error rate at a fixed operating point such as FNMR at a stated FMR. The fourth is the date, because benchmarks move: FaceForensics++ was state of the art in 2019 and is described by the DF40 authors in 2024 as containing "out-of-date forgery techniques from the past four years" (arXiv 2406.13495).

A number with all four fields can be compared with another number that shares them and with nothing else. The published results table on this site carries every row with those fields filled, and the datasets by year timeline shows how fast the dataset field alone has changed. No single benchmark score predicts how a detector performs on manipulations it has not seen.

06Why do benchmark numbers differ between sources?

The same benchmark can produce different numbers in different papers for reasons that have nothing to do with the system under test. The FaceForensics++ paper itself reports XceptionNet at an accuracy of 99.26 percent on uncompressed video and 81.00 percent on the heavily compressed release of the same fakes (arXiv 1901.08971); DeepfakeBench reports Xception at 0.9637 AUC on the c23 release. Different compression, different metric, different preprocessing, different face crops. All of these are "Xception on FaceForensics++" and none is comparable with the others.

Within one paper, prose and tables can also disagree. DeepfakeBench's text gives UCF's within-domain average as 95.37 percent while its Table 3 prints 0.9527; this site uses the table and says so wherever that cell appears. A reader who meets two numbers for one benchmark should look first for a difference in the four fields, then for a transcription difference, before concluding that one system is better than another.