ISO 24614-2:2011

Overview

ISO 24614-2:2011 - Language resource management - Word segmentation of written texts - Part 2: Word segmentation for Chinese, Japanese and Korean - defines rules for segmenting running text into word segmentation units (WSUs) for CJK languages. The standard focuses on delineating WSUs (tokens, words, sub-word units) required for computational processing and is explicitly limited to segmentation - not full morphological or syntactic analysis, nor lexicon design. It documents features common to the three languages and then provides language-specific rules (Chinese, Japanese, Korean), plus terms, markup conventions and examples (e.g., Japanese bunsetsu, Korean eojeol).

Key topics and technical requirements

• Scope and purpose: Applies ISO 24614-1 basic concepts and general principles to CJK word segmentation; restricted to segmentation tasks needed by NLP, IR and MT.
• Common rules: General criteria for identifying WSUs shared across Chinese, Japanese and Korean (compound formation, derivation, fixed expressions, abbreviations, loanwords, foreign-character strings).
• Language‑specific rules: Detailed segmentation rules for:
  • Chinese (including handling of suffixes like 儿 (r) and parts of speech such as nouns, verbs, adjectives, numerals and measure words)
  • Japanese (concepts such as bunsetsu, particles, verb/adjective endings)
  • Korean (eojeol handling, grammatical affixes and agglutination)
• Terminology & markup: Definitions (adnoun, bunsetsu, eojeol, WSU) and recommended markup conventions to annotate segmentation consistently.
• Annex and bibliography: Comparative tables and references to support interoperable annotation.

Practical applications

ISO 24614-2:2011 provides a standardized basis for:

• Training and evaluating tokenizers and segmenters for NLP, machine translation, speech processing, and information retrieval.
• Creating interoperable, annotated corpora and language resources (corpus linguistics, lexicon alignment).
• Improving search indexing, text mining and data preparation in CJK languages where white-space tokenization is not sufficient.
• Ensuring consistent preprocessing in multilingual pipelines that include Chinese, Japanese or Korean.

Who should use this standard

• NLP engineers and data scientists building tokenization and preprocessing modules.
• Corpus linguists, annotators and standards implementers producing CJK annotated datasets.
• MT and IR system developers, lexicographers and localization teams requiring consistent segmentation.
• Academic researchers and software vendors working on language resources for Chinese, Japanese or Korean.

Related standards

• ISO 24614-1:2010 - Basic concepts and general principles (word segmentation)
• ISO 24611 - Morpho-syntactic annotation framework
• ISO 24613:2008 - Lexical markup framework (LMF)
• ISO 24615 - (related lexical/morpho-syntactic standards)

Keywords: ISO 24614-2:2011, word segmentation, Chinese word segmentation, Japanese word segmentation, Korean word segmentation, language resource management, WSU, NLP, machine translation, information retrieval.