From face recognition to deepfake detection: same faces, same pipeline, a different question

This page is the bridge between this site's two sections. It belongs to the deepfake detection hub and points back to the face recognition section, including the ICCV 2021 masked face recognition challenge that this domain once hosted, where the WebFace260M group used exactly this pipeline under a time budget.

01The shared pipeline

Every face system begins with a detector that finds faces in a frame and returns a box and a few landmarks: eyes, nose, mouth corners. The face is then aligned, meaning rotated and scaled so that the landmarks land on fixed positions, and cropped to a standard size. Only then does the main network see it. The DeepfakeBench paper describes this stage for detection in the same terms a recognition paper would: face detection, alignment, cropping, and a fixed number of frames sampled per video, applied identically to every dataset so that differences in the crop do not masquerade as differences between methods (arXiv 2307.01426).

The backbone that follows is often literally the same. The naive detectors in that benchmark are Xception and EfficientNet-B4, image classifiers that recognition systems have also used, and most of the purpose-built detectors in its table keep an Xception backbone and change only what is fed to it or what loss trains it. In recognition the same networks are trained to produce an embedding where two images of one person sit close together; in detection they are trained to produce a score that is high for manipulated faces. The weights differ. The architecture, the input and the preprocessing do not.

02"Who is this?" versus "Is this real?"

The split comes at the final layer and in what the output is compared against. A recognition system's output is only meaningful next to another face: verification asks whether two embeddings belong to the same person, and identification asks which gallery entry a probe is closest to. The system never decides anything about a single image alone. A detector's output is a judgement about a single image or clip, with no second face involved, which is why its errors are counted as false positives and false negatives rather than as false matches and false non-matches.

That difference changes what a benchmark has to provide. A recognition test set needs many images of the same identities so that pairs can be formed; a detection test set needs real and manipulated versions of comparable footage so that the only systematic difference is the manipulation. It also changes what "hard" means. Masks, age and pose make recognition hard because they move an identity's embedding; new generators and compression make detection hard because they remove or alter the traces a detector learned to look for.

03Shared dataset culture and licences

Both fields built their large datasets the same way, by collecting faces of real people from the web, and both distribute them under research licences with an application form. WebFace260M, described on the dataset page, collected about 260 million faces of 4 million identities from the internet and released them under a licence agreement. FaceForensics++ took its 1,000 source videos from YouTube and distributes them by Google form under terms of use; Celeb-DF took 590 videos of public figures from the same site. The public deepfake datasets compared table records the source of faces and the rights position of each set in its own column, because the same consent questions that led to the withdrawal of several recognition datasets apply to the detection sets built on the same material.

One habit did not carry across in full. The Deepfake Detection Challenge dataset was recorded with 3,426 paid actors who agreed to have their likenesses modified (arXiv 2006.07397), a consent model recognition datasets of comparable size never adopted.

04Shared metrics, different operating points

Both fields score a system by sweeping a threshold over its scores and drawing a curve. In recognition the axes are true accept rate and false accept rate, and a result is quoted as TAR at a fixed FAR, often one in ten thousand or one in a million, because the cost of matching two different people is what a deployment must bound. In detection the same curve is called an ROC curve, its axes are true positive rate and false positive rate, and the usual summary is the area under it, AUC, which averages over every threshold rather than fixing one.

The vocabulary differs; the curve is identical, and so is the trap. A single accuracy figure in either field hides the operating point and the class balance of the test set. The face verification metrics page explains the recognition conventions, the metrics page the detection ones, and each links the other for the translation.

05The shared failure mode: winning a benchmark, losing in the wild

Recognition saturated its benchmarks first. Labeled Faces in the Wild, with 13,233 images of 5,749 people and a fixed set of 6,000 pairs, was effectively solved by deep networks trained on web-scale sets, and the field moved to harder protocols and to NIST's evaluations on operational data. Detection is repeating the pattern a decade later and with a sharper cliff. In the DeepfakeBench table ten of the fifteen detectors trained on FaceForensics++ c23 score above 0.93 frame-level AUC on the FaceForensics++ split, and all fifteen score between 0.55 and 0.95 on datasets they never saw; Xception falls from 0.9637 to 0.7365 on Celeb-DF v2. The cross-dataset generalisation page gives the drops model by model.

The lesson recognition learned is the one this site applies to detection: report a score with its test set, protocol and date, treat the in-dataset number as a ceiling, and treat any benchmark as a description of the fakes it contains rather than of the fakes that exist.

06Where the two meet in production: liveness and injection attacks

An identity check at a bank or a border runs both tasks at once. It must recognise the face in front of the camera, and it must decide whether that face is a live person rather than a photograph, a screen or a deepfake injected into the video stream in place of the camera. The second decision is presentation attack detection, tested under ISO/IEC 30107-3 with its own error rates, and it is the place where a face recognition deployment needs a deepfake detector. The liveness and injection attacks page covers how that is tested and where the benchmarks for it stand.