Gender Shades: what the 2018 study measured, how, and what it did not test

This page belongs to the face recognition section, where it sits beside the demographic differentials page on NIST's 2019 measurement of race, sex and age effects in recognition algorithms. The two studies are often quoted together and measure different things.

01What did Gender Shades actually test?

The study evaluated gender classification: a system is shown one face and returns a label, female or male. Three commercial APIs were tested, sold by Microsoft, IBM and Face++ as part of facial analysis bundles, as the paper names them (PMLR 81). The authors note that none of the three "reported performance metrics on existing gender estimation benchmarks in their provided documentation", which is why they built their own test.

The label set was binary. The paper is explicit that the classifiers it tested provide sex labels of female and male, that it uses the same two labels, and that this cannot "adequately capture the complexities of gender or address transgender identities". Ground truth came from each parliamentarian's name, gendered title and photograph.

02Which faces were in the benchmark?

The authors built the Pilot Parliaments Benchmark (PPB) because the two existing face analysis sets they audited were skewed: 79.6% of IJB-A's subjects and 86.2% of Adience's were lighter-skinned. PPB holds one image each of 1,270 members of parliament from three African countries (Rwanda, Senegal, South Africa) and three European countries (Iceland, Finland, Sweden), "selected for gender parity in the national parliaments". The images are official portraits, so pose and lighting are controlled and the faces are public figures with known identities.

Skin type was labelled on the six-point Fitzpatrick scale by three annotators, with a board-certified surgical dermatologist giving the definitive label; types I to III were grouped as lighter and IV to VI as darker. The set is 44.6% female and 55.4% male, 46.4% darker and 53.6% lighter. The four intersectional groups are 21.3% darker female, 25.0% darker male, 23.3% lighter female and 30.3% lighter male.

Counts from Gender Shades, Tables 1 and 3 (PMLR 81, 2018); the IJB-A and Adience darker shares are 100 minus the lighter share the paper gives.

03How was error measured?

The study follows the precedent NIST set for gender classification and reports the true positive rate per group, with the error rate defined as 1 minus TPR, then adds positive predictive value and false positive rate. Every number is a rate on one group of PPB faces, with each classifier queried through its public API in 2017. The headline figures come from Table 4 of the paper.

Error rate in percent (1 minus TPR) on the Pilot Parliaments Benchmark, Gender Shades Table 4, read 9 November 2026.

The paper's own summary of that table: all three classifiers did better on male than female faces, by 8.1 to 20.6 points of error; all three did better on lighter than darker faces, by 11.8 to 19.2 points; all three did worst on darker female faces, with error between 20.8% and 34.7%; and "the maximum difference in error rate between the best and worst classified groups is 34.4%". The overall error rates, 6.3%, 10.0% and 12.1%, would have hidden all of that, which is the study's methodological point: evaluate by intersectional subgroup, not by population average.

04Why is this not a face recognition result?

Face recognition answers a different question with a different metric. Verification compares two faces and reports a false non-match rate at a fixed false match rate; identification searches a gallery and reports false negative and false positive identification rates. Gender Shades never compared two faces and never searched a gallery, so it says nothing about how often a recognition system matches the wrong person. The face verification metrics page defines those error rates, and the how face recognition works page shows where in the pipeline a gender classifier and a recognition model part ways: both start from the same detected, aligned face, but one ends in a two-way label and the other in a 512-dimensional embedding compared against another face.

The recognition evidence on demographics is NIST's. NISTIR 8280 (December 2019) ran 18.27 million images of 8.49 million people through 189 algorithms and found false positive rates that "often vary by factors of 10 to beyond 100 times" across demographic groups, with false negatives varying "often by factors below 3" (NIST.IR.8280). That report measures match errors, not labels, and it covers far more algorithms. The two studies agree in direction, with worse results for women and for darker-skinned people in many systems, and they are not interchangeable: a 34.7% gender misclassification rate is not a 34.7% false match rate, and quoting one as the other misstates both.

05What did the study not measure?

Three limits are stated in the paper and matter when the result is reused. The test set is small, 1,270 images, each a controlled official portrait, so the error rates say nothing about uncontrolled photographs. The classifiers were the API versions available in 2017, and the authors note that the companies described their methods without stating training data, so the result dates a product version rather than a technique. And the groups are lighter against darker on an aggregated Fitzpatrick scale, with the paper noting that the scale itself "is skewed towards lighter skin".

The datasets and consent page records that PPB was built from government portraits posted under non-restrictive licences, a sourcing choice made deliberately after the authors found the existing benchmarks skewed.

06Why does one number per group matter?

Any single accuracy figure is an average over whoever is in the test set, and the test set decides whose errors are counted. Gender Shades showed that with a benchmark 86.2% lighter-skinned, a 34.7% error rate on darker-skinned women barely moves the headline. The same arithmetic applies to any face benchmark, including the NIST evaluations of recognition, which is why NIST measured demographic effects in a report of their own in 2019 and why this site lists the test population next to every score.