Chinese commonly writes with thousands of logographic characters rather than a small alphabet. Characters often behave like words, but individual characters are frequently incomplete units whose meanings depend on neighboring characters; writing also normally omits spaces between words. These properties make Chinese word tokenization a boundary-detection problem rather than a simple whitespace split. section 2.4
Use Jieba segmentation modes according to the downstream task:
- Accurate mode keeps likely word groups together, producing fewer tokens and reducing false word-boundary detections. section 2.4
- Full mode emits all possible word matches, producing more candidate tokens for broad retrieval. section 2.4
- Search-engine mode splits long words into shorter units, supporting compound-word or root-like matching. section 2.4
Segmenting a Chinese sentence
For “西安是一座举世闻名的文化古城” (“Xi’an is a world-famous city for its ancient culture”), accurate segmentation yields 西安 / 是 / 一座 / 举世闻名 / 的 / 文化 / 古城. Full mode additionally exposes smaller matches such as 举世 and 闻名. section 2.4
Listing 2.4 Jieba in accurate mode
>>> import jieba
>>> seg_list = jieba.cut("西安是一座举世闻名的文化古城") #1
>>> list(seg_list)
['西安', '是', '一座', '举世闻名', '的', '文化', '古城']
Run the accurate-mode Python listing to inspect how Jieba converts an unspaced Chinese sentence into word-like tokens. section 2.4
Chinese does not map neatly onto English stemming or lemmatization. Radicals are structural components of characters, and separating them can change meaning; however, some grammatical normalization is comparatively direct, such as removing the plural suffix 们 from 朋友们 to recover 朋友. section 2.4.1
