Overview of Multilingual Embedding and its Algorithm and Implementation

Multilingual Embeddings

The following methods and approaches may be considered to address the challenges of subword-level tokenization

1. adjusting the tokenizer:

To address subword-level tokenization challenges, tokenizer parameters and settings should be appropriately adjusted to customize vocabulary size, tokenization methods, delimiters between tokens, etc., to fit specific tasks and data.

2. reverse tokenization improvements:

Reverse tokenization is important for restoring tokenized text to its original form. Improving the reverse tokenization algorithm, especially to restore proper delimiters between subwords, would be helpful.

3. context model integration:

To address the challenges of subword tokenization, context models could be integrated: transformer models such as BERT and GPT could be used to better understand the tokenized text and modify it to fit the context.

4. processing jargon and unknown words:

In subword tokenization, certain technical terms or unknown words may be split into subwords. To address this issue, a custom unknown word processing mechanism could be introduced to map unknown words to specific subwords. For more details, see also “On Lexical Learning with Natural Language Processing“.

5. Introduction of post-processing:

After tokenization, additional post-processing can be performed for specific tasks. Examples include processing special characters and delimiters, token merging, etc.

6. evaluation and feedback loop:

Set up a feedback loop to evaluate the performance of the tokenizer and make improvements as needed. Check the quality of tokenization on test data and in actual application environments, and make adjustments on an ongoing basis.

7. domain-specific adaptations:

Tailor the tokenizer to specific domains and tasks, taking into account domain-specific vocabulary and special tokenization requirements.

Algorithms used for multilingual embedding

Different algorithms and approaches are used to generate multilingual embeddings. The following are the main algorithms and approaches commonly used to generate multilingual embeddings

1. shared vocabulary and shared embedding:

In this approach, a shared vocabulary of different languages is used to generate a shared embedding vector. Language-independent tokens (e.g., numbers, symbols, etc.) have a common embedding, while language-specific tokens have a unique embedding for each language. This allows text in different languages to be mapped to the same embedding space. For more details, see “Lexical Learning Using Natural Language Processing“.

2. Multimodal Approach:

Multimodal approaches generate embeddings based on multiple data modalities, such as text, audio, and images. This allows for a common language-independent embedding. An example is a model such as BERT, described in “BERT Overview and Algorithms and Example Implementations.

3. transfer learning:

Transfer learning is an approach that takes an embedding learned in one language and applies it to another language, using models trained in a specific language (e.g., Word2Vec described in “Word2Vec“, FastText described in “FastText Overview, Algorithms, and Examples of Implementations“). Global Vectors for Word Representation (GloVe) in “Overview of Glove, Algorithms, and Examples of Implementations“) and apply the embedding to data in different languages. See also “Overview of Transition Learning, Algorithms, and Examples” for details.

4. Multilingual BERT:

BERT (Bidirectional Encoder Representations from Transformers) can be trained as a multilingual model that can be applied to text data in different languages. This allows for the acquisition of multilingual embeddings. For more information, see “BERT Overview, Algorithms, and Example Implementations“.

5. approach using parallel corpora:

Another approach is to use parallel corpora (bilingual data from different languages) to generate embeddings for different languages. This approach generates embeddings based on translations between languages.

6. self-supervised learning:

Self-supervised learning approaches generate embeddings based on word co-occurrence and contextual information. This method automatically generates embeddings from text data in different languages.

7. FastText:

FastText supports data in different languages and generates embedding taking into account subword information. For more information, see “FastText Overview, Algorithms, and Examples of Implementations“.

To generate multilingual embeddings, it is important to choose the right algorithm and approach for the task and data, and to use large data sets and appropriate training methods to obtain high-quality multilingual embeddings.

Examples of Multilingual Embedding Implementations

This section describes how to use the multilingual BERT model as an example of multilingual embedding implementation. Multilingual BERT is available in many languages and is useful for embedding text in different languages. The following is an example Python implementation using the Hugging Face Transformers library.

Installing Hugging Face Transformers: First, install the Hugging Face Transformers library.

pip install transformers

Loading the language BERT model:

Next, load the multilingual BERT model. The following is an example of code that loads a multilingual BERT model (e.g., ‘bert-base-multilingual-cased’).

from transformers import BertTokenizer, BertModel

model_name = "bert-base-multilingual-cased"
tokenizer = BertTokenizer.from_pretrained(model_name)
model = BertModel.from_pretrained(model_name)

Text Tokenization and Embedding:

Now you can tokenize text in different languages and generate embeddings. The following is sample code to tokenize text and obtain embeddings.

text = "Hello, how are you?"

# Tokenize text
input_ids = tokenizer.encode(text, add_special_tokens=True)
input_ids = torch.tensor(input_ids).unsqueeze(0)  # Mini-batch support

# Embedding Generation
outputs = model(input_ids)
embeddings = outputs.last_hidden_state

This code tokenizes the input text and uses the BERT model to generate embeddings.

The Challenges of Multilingual Embedding

Several challenges exist in multilingual embedding. The main challenges are described below.

1. language imbalance:

Multilingual embedding models tend to perform better for some languages than others. This is due to imbalance in training data and lack of resources, and focusing on some major languages may degrade the performance of other languages.

2. application to low-resource languages:.

The performance of multilingual embedding may be inadequate for low-resource and minority-speaker languages. Insufficient training data or inadequate adaptability of the model may be a problem.

3. word sense ambiguity:

Multilingual embedding may not be able to distinguish word senses when a single word has multiple meanings. Especially in the case of polysemous words, it is difficult to identify the exact sense of a word.

4. translation consistency:

Although the goal of multilingual embedding is to place text in different languages in the same vector space, translation consistency may not be guaranteed. That is, text with the same meaning may be mapped to different embeddings across different languages. For more information, see “Overview of Translation Models, Algorithms, and Examples of Implementations.

5. dealing with unknown words:

Multilingual embedding cannot generate appropriate embeddings for unknown words that are not present in the training data. Mechanisms are needed to deal with unknowns.

6. different language families:

Consistency of embeddings across unrelated languages can be difficult. For example, it is difficult to ensure consistency between Indo-European and Uralic languages.

7. task-dependent:

Multilingual embedding may not be suitable for certain NLP tasks. Task-specific embeddings are needed.

To address these challenges, quality improvement of multilingual embeddings and training of custom models focused on specific languages are being undertaken. In addition, resource augmentation, support for low-resource languages, and integration of task-specific embeddings are being considered as ways to address the challenges.

Addressing the Challenges of Multilingual Embedding

The following methods and measures can be considered to address the challenges of multilingual embedding

1. augment training data:

Increasing the amount and diversity of training data in multiple languages will help improve the performance of multilingual embedding. Enriching training data with additional corpora and data from different genres will be important.

2. attention to low-resource languages:

Focus on low-resource languages and enhance data collection and training specifically for those languages. Improved attention to low-resource languages will improve the equilibrium of multilingual embedding.

3. transfer learning:

Build on existing multilingual embedding models and consider how to generate embeddings appropriate for specific tasks and domains. Fine tuning to adjust pre-trained embeddings to task-specific embeddings would be useful.

4. interlanguage measures:

Interlanguage measures can be used to effectively measure and maintain the consistency of embeddings in different languages. For example, Procrustes analysis can be used to align the embeddings of different languages.

5. unknown word handling:

It is important to incorporate measures to deal with unknown words. Consider ways to split unknown words into subwords, e.g., using subword tokenizers (e.g., SentencePiece as described in “Overview of SentencePiece with Algorithm and Example Implementation“, WordPiece as described in “Overview of WordPiece with Algorithm and Example Implementation“). We will consider how to split unknown words into sub-words.

6 Task-specific embeddings:

Leverage task-specific data and domain information to generate embeddings appropriate for specific tasks. Task-specific embeddings can be trained to improve performance.

7. resource sharing and collaboration:

It will be important to share resources and collaborate to improve the quality of multilingual embeddings. Collaboration with research institutions and communities will enable continuous improvement of multilingual embeddings.

Reference Information and Reference Books

For more information on natural language processing in general, see “Natural Language Processing Technology” and “Overview of Natural Language Processing and Examples of Various Implementations.

Reference books include “Natural language processing (NLP): Unleashing the Power of Human Communication through Machine Intelligence“.

“Practical Natural Language Processing: A Comprehensive Guide to Building Real-World NLP Systems“

“Natural Language Processing With Transformers: Building Language Applications With Hugging Face“