Information technology — Biometric performance testing and reporting — Part 10: Quantifying biometric system performance variation across demographic groups
Introduction: As the use of biometric technology increases, so too does public interest in establishing whether the technology performs similarly for all individuals. Stakeholders are asking government and industry organizations that use biometric technology to establish whether these technologies vary in performance for different demographic groups. The intention of this document is to provide guidance on how to measure and report performance variation across demographic groups.[2] This document is intended to help organizations evaluate demographic performance in biometric systems and report their results. Specifically, this document outlines how to measure and report biometric performance variations across demographic groups. It provides a set of metrics and best practices to facilitate such testing. However, this document does not provide guidance on how to establish specific causes for the observed variations. The following demographic variables are explicitly discussed in this document:[7][10][12] — biological characteristics, such as: — sex, age, weight, height and skin lightness; — social constructs, such as: — ethnicity, gender and language. Many other variables can cause systematic changes in biometric characteristics or in how individuals interact with biometric systems. The following demographic variables are relevant although not explicitly discussed in this document: — performance variations based on temporary states, such as: — self-styling (e.g. makeup, eyewear, mask-wearin
Scope: This document establishes requirements for estimating and reporting on performance variations observed when cohorts belonging to different demographic groups engage with biometric enrolment and recognition systems. In this context, performance refers to failure-to-enrol rate, failure-to-acquire rate, shifts in comparison score, recognition error rates, and aspects of response and processing time (throughput). This document is applicable to the following: — demographic group membership; — using phenotypic measures; — reporting on tests; — stating statistical uncertainty estimates; — operational thresholds settings; — equitability; — procurement agency activities. This document also provides terms and definitions to be used when reporting performance variation across demographic groups. This document is applicable to: — technology evaluations of algorithms, subsystems and systems; — scenario evaluations of systems; — operational evaluations of fielded systems. Application of this document does not require detailed knowledge of a system’s algorithms but it does require specific knowledge of the demographic characteristics for the population of interest.