Artificial Intelligence — Evaluation methods for accurate natural language processing systems
Introduction: This document focuses on the evaluation of natural language processing (NLP) systems, as regards their functional suitability as defined in ISO/IEC 25059:2023 [1]. This includes functional correctness (often informally referred to as ‘performance’ by NLP practitioners, which ISO/IEC 25059:2023 [1], Annex C discourages due to ambiguity, or 'accuracy' in some regulation, which can be inconsistent with technical usage), as well as other quality characteristics such as functional adaptability. While other standards exist on the evaluation of AI systems, such as ISO/IEC DIS 4213, they only contain generic metrics, targeting the evaluation of some base AI tasks like classification, and those metrics are either not sufficiently specified to be applied in turn on NLP systems or not applicable to part of these NLP systems. This document therefore builds on ISO/IEC DIS 4213 to provide the NLP-specific complements that are necessary to enable the evaluation of NLP systems in particular. This includes additional specifications on generic metrics used beyond NLP, as well as NLP-specific evaluation methods, encompassing both automated and human evaluation. © ISO/IEC 2026 – All rights reserved
Scope: This document specifies the evaluation of natural language processing systems, in the sense of measuring the quality of a system’s results to assess its functional suitability. It provides a definition of evaluation methods for those systems, together with guidance on how to select, implement and interpret those evaluation methods. This document covers quantitative metrics as well as other evaluation methods. It includes requirements on the implementation of the described metrics, and further requirements on the technical resources involved in the evaluation process.