Overview
ISO/IEC TR 29170-1:2017 provides guidelines and best practices for evaluating image coding systems (codecs) for images and image sequences. The technical report defines a common vocabulary, classifies evaluation methods, and recommends procedures for testing image quality and codec behavior. It addresses ordinary and unusual imagery (for example, high dynamic range (HDR) and high colour depth) while explicitly excluding hardware complexity assessment.
Key topics
- Evaluation categories:
- Subjective assessment (human observers, mean opinion score - MOS)
- Objective assessment (computational image-quality metrics)
- Computational assessment (algorithmic/complexity measures)
- Note: hardware complexity (real-time performance, implementation cost) is outside the scope.
- Terminology & definitions: common terms such as codec, channel, component bit depth, pixel, quality loss, generational quality loss, drift, constant/variable bit rate.
- Test image selection & characteristics: guidance on selecting representative images and describing image properties (channels, sample precision, resolution).
- Compression performance metrics: bits per pixel (bpp, defined as 8·L / (w·h)), compression ratio (CR), and considerations for variable vs. constant bit-rate systems.
- Subjective testing best practices: observer selection (expert vs non-expert), number of observers, visual acuity, instructions, evaluation scales, viewing conditions (references to ISO 3664 and ISO 9241), and statistical analysis.
- Objective & computational metrics: reference to common measures and algorithms (examples found in annexes: PSNR, MSE, SSIM/MS-SSIM, CIEDE2000, HDR-VDP and other visual difference predictors).
- Special cases: handling HDR, high colour depth imagery, error resilience, recursive compression effects and verification of codec characteristics (see informative annexes A–D).
Applications and who uses it
ISO/IEC TR 29170-1 is useful for:
- Codec developers and researchers - to design and validate compression algorithms and measure visual quality.
- Image processing and multimedia QA teams - to establish reproducible test procedures and pass/fail criteria.
- Broadcast, streaming and imaging device manufacturers - to evaluate perceptual quality under different bit rates and content types including HDR.
- Standards bodies and test-lab operators - to harmonize vocabulary, test image selection, and assessment approaches.
- Tool vendors - to implement objective and computational metrics consistent with standard guidance.
Related standards
- ISO 3664 (viewing conditions for image evaluation)
- ISO 9241 (ergonomics of human-system interaction / display requirements)
- Referenced metrics and codec definitions tie into broader JTC 1 / SC 29 image coding work.