Face BenchmarkHow face recognition and deepfake detection are measured, from the papers.
Menu

Evaluate a deepfake detector on your own data: a short, fair protocol

The Face Benchmark editorsPublishedFigures checked

A published detector score describes the fakes in the benchmark it was measured on, at that benchmark's compression, under its protocol, on a date. Your fakes, your compression and your false positive budget are different, so the published number is not your number. Five steps make your own measurement comparable with the published rows instead of a second kind of unmeasured claim: fix the question, build a held-out set, match a protocol, record raw scores, and report the value with every condition attached.

STPThe protocol

Five steps, in order. Each ends with one line you can tick before moving on.

  1. DEFINE

    Step 1: Fix the question and the operating point

    Write down what counts as a fake for your purpose, which kinds of manipulation you care about (swap, reenactment, synthesis, editing), and the false positive rate you can tolerate on genuine material. Choose the metric now: AUC if you want the whole curve, true positive rate at your fixed false positive rate if you want the number a deployment will live with. Tick when the question, the kinds and the operating point are written down before any clip is scored.

  2. BUILD

    Step 2: Build a held-out test set

    Collect real and fake clips the detector has never seen, from sources and generation methods you log, and in the compression your use will actually meet. Keep the share of fakes close to what you expect in use, because accuracy and precision move with it. Check for overlap with the detector's training data by source video, not only by file: FaceForensics++, Celeb-DF and the DFDC set are described on the datasets page so you can exclude them. Tick when every clip has a logged source, method and compression and none overlaps a training set.

  3. MATCH

    Step 3: Match a published protocol

    Pick the published result you want to compare with and copy its conditions: metric, frame or video level, how frame scores are aggregated, face crop and alignment, number of frames sampled per clip. DeepfakeBench fixes all of these in configuration and ships the preprocessing, so running your set through its pipeline makes your number comparable with its table. Tick when you can state the matched conditions in one sentence of the form the results page uses.

  4. RUN

    Step 4: Run and record

    Score every clip and save the raw per-frame or per-clip scores, not only the summary, so the curve can be redrawn at another threshold later. Set the threshold from step 1 before looking at results on the test set; a threshold tuned on the test set is a result about the test set. Tick when the raw scores are saved with the clip identifiers and the threshold was fixed in advance.

  5. REPORT

    Step 5: Report the number with its conditions

    Report the value beside the train set the detector came with, your test set and its composition, the protocol (unseen or not, compression, level), the metric and threshold, and the date, in the same form as the rows on the results page. Report the in-dataset score the detector's paper claims next to it, so the drop is visible. Tick when a reader could reproduce the number from your report without asking you anything.

Protocol flowchartFive boxes linked top to bottom by arrows: Fix the question and operating point, then Build a held-out test set, then Match a published protocol, then Run and record per-clip scores, then Report the value with its conditions. A dashed path returns from step 4 to step 2 if a test clip overlaps with training data.1Fix the question and operating point2Build a held-out test set3Match a published protocol4Run and record per-clip scores5Report the value with its conditionsIF A CLIP LEAKED, REBUILD
Figure 1The protocol as a flow. Each box is one step above. The dashed red path returns to step 2 if any test clip turns out to overlap with training data.

This protocol is part of the deepfake detection hub. It is the practical side of how to read a detection paper: the four fields you look for in a paper are the four fields your own report must state.

LIMLimits

  • L1A fair test on your data measures the detector on your data today. New generators and new platform encodings change the answer, as the cross-dataset generalisation and in-the-wild results show.
  • L2Small test sets give wide intervals: a difference of a few AUC points between two detectors on a few hundred clips is not evidence of a difference, and the metrics page explains why a single accuracy figure says less still.
  • L3Matching a published protocol exactly is often impossible without the authors' code and weights; where you cannot match a condition, report the mismatch rather than the comparison.
  • L4The protocol tests detection of manipulated media. It does not test the capture path, which the liveness page covers, and it says nothing about any particular clip.

FAQQuestions

Why can I not use the accuracy figure from the detector's paper?

Because it was measured on the paper's test set, at its compression, under its protocol and class balance. The FaceForensics++ paper shows the same detector at an accuracy of 99.26% on raw video and 81.00% at heavy compression on the same fakes; your material sets the conditions, so you have to measure.

How many clips do I need?

Enough that the difference you care about is larger than the uncertainty of the estimate. There is no fixed number; report the count of real and fake clips with the result, and treat small differences on small sets as noise.

Should I fine-tune the detector on my data first?

Only if you then test on a held-out part your fine-tuning never saw, and report both numbers. Deepfake-Eval-2024 reports an off-the-shelf AUC of 0.63 and a fine-tuned AUC of 0.82 for the same model as two separate results, because they answer different questions.

How do I make my result comparable with DeepfakeBench rows?

Run your test set through the DeepfakeBench preprocessing and evaluation code with the same frame sampling and metric, and state that you did. The DeepfakeBench page describes what the benchmark fixes; the results page shows the form a comparable row takes.

SRCSources

  1. S1Checked
  2. S2Checked
  3. S3Checked