Overview
ISO/IEC TR 24027:2021 is a Technical Report from ISO/IEC that addresses bias in AI systems with a particular focus on AI-aided decision-making. It describes measurement techniques and methods for assessing bias and provides guidance to identify, evaluate and treat bias-related vulnerabilities across the entire AI system lifecycle - from data collection and design through training, continual learning, testing, deployment and use. The report is intended to help practitioners understand sources of unwanted bias and apply systematic risk-reduction practices.
Key Topics and Technical Highlights
- Sources of bias: human cognitive biases (e.g., implicit, confirmation, group attribution), societal bias, data bias (sampling, labelling, processing), and engineering decisions (feature engineering, algorithm selection, hyperparameter tuning).
- Assessment methods: foundational tools such as the confusion matrix and a set of fairness metrics discussed in the report - including equalized odds, equality of opportunity, demographic parity, and predictive equality - along with other applicable metrics.
- Lifecycle treatment: practical guidance for handling bias during inception (requirements and stakeholder identification), design and development (data representation, training, adversarial mitigation), verification and validation (static data analysis, label checks, internal/external validity testing), and deployment (continuous monitoring and transparency tools).
- Supporting material: informative annexes offering examples of bias, related open-source tools for bias assessment/mitigation, and a mapping example to ISO 26000 (social responsibility).
Practical Applications - Who Should Use It
ISO/IEC TR 24027:2021 is relevant to:
- Data scientists & ML engineers designing models and preparing datasets to reduce unwanted bias.
- Product managers & system architects setting requirements for fair AI-aided decision-making.
- AI auditors, compliance officers and risk managers evaluating fairness, transparency and governance controls.
- Regulators and procurement teams assessing vendor claims about bias mitigation and model validation.
- QA and validation teams implementing testing, monitoring and continuous validation pipelines.
Use cases include fairness testing in hiring systems, credit scoring, healthcare triage tools, automated decision support, and any domain where biased outcomes create ethical, legal or reputational risk.
Related Standards and Resources
- Produced by ISO/IEC JTC 1/SC 42 (Artificial Intelligence) - aligns with broader AI governance work.
- Annex C maps themes to ISO 26000 (social responsibility).
- Annex B lists open-source tools for measurement and mitigation of bias.
ISO/IEC TR 24027:2021 is a practical reference to help organizations operationalize bias assessment and mitigation across AI lifecycles, improve AI fairness, and build more transparent, accountable AI systems.