PD ISO IEC TS 25058-2024 PDF

Full title and description

ISO/IEC TS 25058:2024 — Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — Guidance for quality evaluation of artificial intelligence (AI) systems. This Technical Specification provides guidance on evaluating AI systems using an AI system quality model and is intended for organisations involved in the development, deployment and use of AI systems.

Abstract

This Technical Specification gives guidance for evaluation of artificial intelligence (AI) systems using an AI system quality model. It describes evaluation objectives, recommended evaluation approaches and how to apply quality models to judge AI system quality in a range of contexts. The document is applicable to all types of organisations engaged in the development and use of AI.

General information

• Status: Published (Technical Specification, to be reviewed under ISO review cycle).
• Publication date: January 2024 (published 24 January 2024, edition 1).
• Publisher: ISO/IEC (ISO/IEC JTC 1, developed with input from SC 42).
• ICS / categories: 35.080 — Software engineering / SQuaRE.
• Edition / version: Edition 1 (2024) — Technical Specification ISO/IEC TS 25058:2024.
• Number of pages: 20 pages.

Scope

Provides guidance for evaluating the quality of AI systems by applying an AI-specific quality model. It supports selection of evaluation objectives, tailoring of evaluation methods to AI characteristics (e.g., probabilistic outputs, learning behaviour, data dependence) and use of quality models to structure assessments across stakeholders and contexts of use. The specification is intended to be technology-agnostic and applicable across industry sectors.

Key topics and requirements

• Guidance on applying an AI system quality model to structure evaluations (linking requirements, quality characteristics and evaluation tasks).
• Recommendations for defining evaluation objectives and selecting appropriate evaluation approaches for AI systems (including empirical tests, benchmarks, and contextual assessments).
• Alignment with the AI quality model developed in ISO/IEC 25059 (characteristics and sub-characteristics tailored to AI such as functional adaptability, robustness, transparency and intervenability).
• Notes on measurement: the TS offers guidance and indicators but does not fully replace forthcoming measurement standards; measurement-focused work (AWI 25058) is under development to expand measurement guidance.
• Practical considerations for contextualising quality-in-use, sampling and test design, and combining quantitative and qualitative evidence in AI quality evaluation.

Typical use and users

Intended users include AI developers and engineers, quality-assurance and test teams, independent evaluators, acquirers/procurement teams, risk and compliance officers, and policy/regulatory stakeholders who need structured guidance for assessing AI system quality across development, deployment and in-use phases. The guidance is suitable for organisations of all sizes and sectors.

Related standards

Part of the ISO/IEC SQuaRE family and closely related to: ISO/IEC 25059 (quality models for AI systems) — provides the AI-specific quality model used by this TS; ISO/IEC 25010 (product quality model) — the general product-quality framework; and ISO/IEC 2502x measurement standards (e.g., 25022/25023 family) for product and quality-in-use measurement. An Approved Work Item (AWI 25058) is under development to provide extended measurement and guidance and will replace this TS in time.

Keywords

AI system quality, evaluation guidance, SQuaRE, ISO/IEC TS 25058, AI quality model, measurement guidance, robustness, transparency, functional adaptability, intervenability, quality-in-use.

FAQ

Q: What is this standard?

A: ISO/IEC TS 25058:2024 is a Technical Specification that provides guidance for evaluating the quality of artificial intelligence systems using an AI system quality model. It helps organisations structure evaluation objectives and choose suitable evaluation methods.

Q: What does it cover?

A: It covers guidance on applying an AI quality model to evaluations, suggested evaluation approaches (empirical and contextual), how to relate evaluation results to quality requirements, and practical considerations for planning and conducting AI quality assessments. It does not replace detailed metric standards but guides how to use quality models in evaluation.

Q: Who typically uses it?

A: AI developers, QA/test teams, independent evaluators, procurement and acquisition teams, risk and compliance professionals, and anyone responsible for assessing or assuring the quality of AI systems.

Q: Is it current or superseded?

A: As published in January 2024 it is current as a Technical Specification. An Approved Work Item (AWI 25058) is under development to extend measurement guidance and is intended to replace or supersede the TS in a future revision. The TS will be subject to ISO's normal review process.

Q: Is it part of a series?

A: Yes — it is part of the ISO/IEC 25000 SQuaRE series and is intended to be used alongside related SQuaRE documents such as ISO/IEC 25059 (AI quality models), ISO/IEC 25010 (product quality model) and the ISO/IEC 2502x measurement standards.

Q: What are the key keywords?

A: AI quality, quality evaluation, SQuaRE, AI system quality model, measurement guidance, robustness, transparency, functional adaptability, intervenability.