tokenization

美 　英

• 網(wǎng)絡(luò)標記化；斷詞；符號化

例句

There's no need for a tokenization phase, and lookahead assertions can be used for a limited degree of context-sensitivity.

沒有必要再進行符號化解析，或者用于有限度上下文敏感的前向斷言。

By the way, as you will see in the examples, DParser uses no separate tokenization pass, just straight parsing.

順便提一句，如您將在示例中所看到的，DParser不使用任何單獨的標記傳遞，而只是直接解析。

User input may be used to adjust filtering and tokenization of the messages.

用戶輸入的內(nèi)容可能會用于校正過濾和用于標記消息。

It supports natural language processing like word stem reduction and tokenization in multiple languages.

它支持自然語言處理，比如多種語言中的詞根縮減和標記化。

Document tokenization, which is the phase that transforms a binary document into plain text, containing most of the document's content

文檔標記化，它是將二進制文檔轉(zhuǎn)換為包含文檔的大部分內(nèi)容的純文本的階段

A Study About Chinese Dialect Identification Based on Tokenization and Language Model

基于符號化和語言模型方法的漢語方言自動辨識