Subword-level tokenization

Subword-level tokenization

Subword-level tokenization is a natural language processing (NLP) approach that breaks text data into subwords (subwords) smaller than words. This is used to facilitate understanding of the meaning of a sentence and to alleviate lexical constraints. There are several approaches to subword-level tokenization, the most common of which include

1. Byte Pair Encoding (BPE):

BPE, described in “Overview of Byte Pair Encoding (BPE) and Examples of Algorithms and Implementations” is an effective algorithm for splitting text into subwords. By merging high-frequency characters or subwords to create new subwords, the vocabulary is effectively compressed.

2. WordPiece:

WordPiece, described in “Overview of WordPiece and Examples of Algorithms and Implementations” is used in Google’s BERT model and is a similar approach to BPE. WordPiece generates tokens by combining high frequency letters or sub-words and includes the most common sub-words in the vocabulary to deal with unknown words.

3 Unigram Language Model Tokenizer:

The Unigram Language Model Tokenizer, described in “Overview of the Unigram Language Model Tokenizer with Algorithm and Example Implementation” is a method for selecting tokens based on the frequency of subwords in the lexicon. High-frequency subwords become tokens as they are, while low-frequency subwords may be split as subwords.

4. SentencePiece:

SentencePiece, described in “SentencePiece Overview, Algorithm, and Example Implementation” is a multilingual subword tokenizer with an easy to learn tokenization model that can be applied to a variety of languages. User-configurable parameters control the tokenization method.

The advantages of subword-level tokenization include:

• Eased vocabulary constraints: If word-level tokenization has severe vocabulary constraints, subword tokenization can handle a larger vocabulary, allowing for a wider variety of languages and unknown words.
• Multilingual: Subword tokenization can be applied to multiple languages, making it easier to maintain tokenization consistency across languages.
• Handling unknown words: Subword tokenization is also effective for unknown words, allowing known subwords to be combined to generate new tokens.
• Preserving part-of-speech information: Subword tokenization usually makes it easier to preserve part-of-speech and pragmatic information within a word.

On the other hand, challenges with subword-level tokenization include the additional cost of preprocessing and the complexity of the inverse operation, as well as the need for adjustments related to training and selection of tokenization models to improve tokenization accuracy in general.

Example implementation of subword-level tokenization

To implement subword-level tokenization, you can use common tools and libraries or implement the algorithm on your own. Below is an example implementation of subword-level tokenization using SentencePiece, a popular tool.

Installing SentencePiece: First, install SentencePiece.

pip install sentencepiece

SentencePiece training: Use SentencePiece to train subword tokenizers. Prepare an appropriate corpus as training data. In this example, we use an English corpus.

import sentencepiece as spm

# Loading of training data
input_file = "english_corpus.txt"

# SentencePiece model training
spm.SentencePieceTrainer.Train(f"--input={input_file} --model_prefix=subword_model --vocab_size=8000")

The code learns a subword model from “english_corpus.txt” and sets the vocabulary size to 8000.

Tokenize: Tokenize text data using the learned subword model.

sp = spm.SentencePieceProcessor()
sp.Load("subword_model.model")

text = "This is an example sentence."
tokens = sp.EncodeAsPieces(text)

The EncodeAsPieces method can be used to obtain a list of tokens that split the text data into subwords.

Reverse Tokenization: Tokenized text can be restored to its original form by performing reverse tokenization.

detokenized_text = sp.DecodePieces(tokens)

The DecodePieces method is used to restore the tokenized text to its original form.

This would be a basic example implementation of subword-level tokenization using SentencePiece. Other tools and libraries can also be used, such as Subword-nmt and the tokenizer from the Hugging Face Transformers library. It will also be important to adjust the parameters and settings to tune the tokenizer and adapt it to specific tasks.

The challenges of tokenization at the subword level

Several challenges exist with subword-level tokenization. They are described below.

1. loss of context:

In subword-level tokenization, words are sometimes broken into subwords. This can result in the loss of some contextual information and can make it difficult to understand the context, especially when long words or phrases are split into subwords.

2. difficulties with reverse tokenization:

It can be difficult to restore text that has been tokenized at the subword level to its original form. Reverse tokenization requires restoring whitespace and delimiters between tokens, making the handling of delimiters between subwords a particular challenge.

3. subword discontinuity:

Subword tokenization can cause a single word to be split into multiple subwords. This can result in fragmented word meanings, which can limit some NLP tasks.

4. tokenizer alignment:

It will be important to adjust the lexical size of subword tokenization and the hyperparameters of the tokenizer. Finding an appropriate vocabulary size is difficult.

5 Dependence on training data:

The subword model is dependent on training data. To apply it to new text data, the model needs to be re-trained from the training data.

6. contextual misunderstanding:

Subword tokenization has no information about context and may produce tokenization results that depend on different meanings and contexts of words.

7. suitability for a specific task:

While subword-level tokenization is generally suitable for many NLP tasks, there is a need to tailor tokenizers to specific tasks and domains.

A variety of measures are needed to address these challenges, including tokenizer configuration and adjustment, contextual completion, improved reverse tokenization, and customization to specific NLP tasks. Subword tokenization is a flexible and powerful tool, but proper coordination and evaluation are essential.

Addressing the issue of subword-level tokenization

The following methods and approaches may be considered to address the challenges of subword-level tokenization

1. adjusting the tokenizer:

To address subword-level tokenization challenges, tokenizer parameters and settings should be appropriately adjusted to customize vocabulary size, tokenization methods, delimiters between tokens, etc., to fit specific tasks and data.

2. reverse tokenization improvements:

Reverse tokenization is important for restoring tokenized text to its original form. Improving the reverse tokenization algorithm, especially to restore proper delimiters between subwords, would be helpful.

3. context model integration:

To address the challenges of subword tokenization, context models could be integrated: transformer models such as BERT described in “Overview of BERT and examples of algorithms and implementations” and GPT described in “Overview of GPT and examples of algorithms and implementations” could be used to better understand the tokenized text and modify it to fit the context.

4. processing jargon and unknown words:

In subword tokenization, certain technical terms or unknown words may be split into subwords. To address this issue, a custom unknown word processing mechanism could be introduced to map unknown words to specific subwords. For more details, see also “On Lexical Learning with Natural Language Processing“.

5 Introduction of post-processing:

After tokenization, additional post-processing can be performed for specific tasks. Examples include processing special characters and delimiters, token merging, etc.

6. evaluation and feedback loop:

Set up a feedback loop to evaluate the performance of the tokenizer and make improvements as needed. Check the quality of tokenization on test data and in actual application environments, and make adjustments on an ongoing basis.

7. domain-specific adaptations:

Tailor the tokenizer to specific domains and tasks, taking into account domain-specific vocabulary and special tokenization requirements.

Reference Information and Reference Books

For more information on natural language processing in general, see “Natural Language Processing Technology” and “Overview of Natural Language Processing and Examples of Various Implementations.

Reference books include “Natural language processing (NLP): Unleashing the Power of Human Communication through Machine Intelligence“.

“Practical Natural Language Processing: A Comprehensive Guide to Building Real-World NLP Systems“

“Natural Language Processing With Transformers: Building Language Applications With Hugging Face“