Overview of neural ranking models, algorithms and implementation examples.

Mathematics Machine Learning Artificial Intelligence Graph Data Algorithm Programming Digital Transformation Algorithms and Data structures Navigation of this blog

Overview of neural ranking models

Neural ranking models are a type of machine learning model used in search engines and recommendation systems, where the main objective is to sort items (e.g. web pages, products, etc.) in the best ranking based on a given query or user information.

For a typical search engine, it is important to display first the web pages that are most relevant to the user’s query, and to achieve this, the search engine considers a number of factors to determine the ranking of web pages. These include keyword match, page credibility and the user’s previous click history.

Neural ranking models have attracted attention as a way to effectively combine these factors for ranking, and a typical model is built using the following steps.

1. feature extraction: quantifying the characteristics of a query or page. This includes the frequency of occurrence of keywords, the content of certain parts of the page, user history, etc.

2, Feature Combination: combines the extracted features and assigns a weight to each feature. This adjusts the degree of influence of each feature on the ranking.

3, Neural Network Training: train a neural network using the features and weights created above as input. This trains a model for optimal ranking.

4. Ranking Inference: Infer a ranking for a new query or page using the trained model. This generates the final ranking result.

The advantage of neural ranking models is that they can effectively combine multiple features to capture complex relationships, an approach that can also be trained on large data sets to understand sophisticated patterns and trends and provide better rankings.

Algorithms associated with the neural ranking model.

There are various algorithms and methods for neural ranking models. Some of the most representative ones are described below.

1. RankNet: RankNet was one of the first algorithms to use neural networks to learn rankings, and was proposed by Microsoft Research in 2005.RankNet takes ranked pairs as input and the order of those pairs is learning.

Specifically, the learning procedure is as follows.

  1. Calculate the loss per pair and adjust the weights of the network to minimise that loss.
  2. It learns to generate small losses when the order of the pairs is correct and large losses when the order is reversed.

2. lambdaMART: LambdaMART can be a gradient boosting algorithm using decision trees for ranking. To solve the ranking problem, a ranking function is learnt and LambdaMART uses the concept of ranked pairs to determine if the order is correct.

3. ListNet: the ListNet will be the one that learns to minimise the loss of the entire ranked list. Specifically, it uses the softmax function to calculate the ranking probabilities for each item and learns to maximise those probabilities.

4. Neural Sort: Neural Sort aims to learn permutations representing order, and unlike RankNet, which is based on pairwise comparisons, Neural Sort learns permutation functions directly and uses them for ranking.

5. BERT (Bidirectional Encoder Representations from Transformers): BERT is a language model widely used in the field of natural language processing, but also applied to ranking tasks. pairs as input and determine their relevance for ranking.

Application of the neural ranking model.

Neural ranking models have been widely applied in various domains, and examples of their application are described below.

1. search engines: search engines such as Google and Bing use neural ranking models to rank search results. They learn the relevance of the user’s query to the search results and display the results in the most appropriate ranking.

2. product recommendations: online retailers and streaming services (e.g. Netflix, Amazon, Spotify) use neural ranking models to recommend individually customised products and content based on users’ past purchase and viewing history.

3. ad display: online advertising platforms use neural ranking models to rank ads based on user interests and behaviour. This enables more effective adverts to be displayed, improving click and conversion rates.

4. improving web search results: web search engines use neural ranking models to improve the quality of search results. For example, they are used to show pages with more relevant information higher up and to weed out spam and low-quality content.

5. medical diagnostic assistance: in the field of medicine, research is being conducted on the use of neural ranking models to provide assistance in the diagnosis of diseases, using patient symptoms and test results as input. This enables appropriate treatment to be initiated at an earlier stage.

6. recommendation systems: online platforms and applications can analyse a user’s behavioural history and preferences and use neural ranking models to make recommendations. Examples include YouTube video recommendations and social networking post displays.

7. music and film recommendations: music streaming services and film distribution platforms learn users’ preferences and past choices and use neural ranking models to recommend appropriate songs and films.

8. real-time in-game item display: online games use neural ranking models to optimise the display of items and enemy characters in real-time according to player behaviour and play style.

Example implementation of a neural ranking model.

The following describes an example implementation of a neural ranking model using PyTorch.

1. example implementation of RankNet: RankNet becomes a neural ranking model that learns the order of ranked pairs.

import torch
import torch.nn as nn
import torch.optim as optim

class RankNet(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(RankNet, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_size, 1)
        self.sigmoid = nn.Sigmoid()

    def forward(self, x1, x2):
        out1 = self.relu(self.fc1(x1))
        out2 = self.relu(self.fc1(x2))
        score1 = self.sigmoid(self.fc2(out1))
        score2 = self.sigmoid(self.fc2(out2))
        return score1, score2

# Instantiating the model
model = RankNet(input_size=10, hidden_size=64)

# Loss functions and optimisers
criterion = nn.BCELoss()  # binary cross entropy error
optimizer = optim.Adam(model.parameters(), lr=0.001)

# Data preparation
x1 = torch.randn(32, 10)  # Data on the first of the ranked pairs.
x2 = torch.randn(32, 10)  # Second data for ranked pairs.
labels = torch.randint(0, 2, (32,))  # Labels in the correct sequence
# learning
for epoch in range(10):
    optimizer.zero_grad()
    score1, score2 = model(x1, x2)
    loss = criterion(score1 - score2, labels.float())
    loss.backward()
    optimizer.step()
    print(f"Epoch [{epoch + 1}/10], Loss: {loss.item()}")

 

2. example implementation of LambdaMART: LambdaMART is a gradient boosting algorithm for ranking. The following is an example of implementing LambdaMART using the LightGBM library.

import lightgbm as lgb
from sklearn.datasets import load_svmlight_file

# Loading data
X_train, y_train = load_svmlight_file('train.txt')
X_val, y_val = load_svmlight_file('val.txt')

# Conversion of datasets to LightGBM format.
lgb_train = lgb.Dataset(X_train, y_train)
lgb_val = lgb.Dataset(X_val, y_val)

# Parameter settings
params = {
    'boosting_type': 'gbdt',
    'objective': 'lambdarank',
    'metric': 'ndcg',
    'num_leaves': 31,
    'learning_rate': 0.1,
    'max_depth': -1,
    'min_child_samples': 20,
    'subsample': 0.8,
    'subsample_freq': 1,
    'colsample_bytree': 0.8,
    'verbose': -1
}

# Learning the model
gbm = lgb.train(params, lgb_train, num_boost_round=100, valid_sets=[lgb_train, lgb_val], early_stopping_rounds=10)

# Prediction on test data.
X_test, y_test = load_svmlight_file('test.txt')
preds = gbm.predict(X_test)

print(preds)

3. example implementation of ranking using BERT: Implementing a ranking model using BERT can be easily done using the Transformers library from Hugging Face.

from transformers import BertTokenizer, BertForSequenceClassification
import torch

# Loading models and tokenisers
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Text encoding and padding
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt", padding=True, truncation=True)

# 推論
outputs = model(**inputs)
logits = outputs.logits

print(logits)
Challenges and countermeasures for neural ranking models.

Although the neural ranking model is a powerful and effective method, there are several challenges. These challenges and measures to address them are described below.

1. biased or missing data:

Challenge: the ranking task requires labelled data for query and item pairs, but this data may be unbalanced or insufficient.
Solution:
Sampling innovations: use appropriate sampling strategies to balance the data (e.g. oversampling or undersampling to unbalanced data).
Generation-based approaches: use generative models to synthesise additional ranking data.
Reinforcement learning: apply reinforcement learning to improve ranking policies.

2. labelling uncertainty:

Challenge: user queries and choices are sometimes ambiguous or ambiguous, and the correct ranking may not be clear.
Solution:
Probabilistic models: represent uncertainty by treating ranking scores probabilistically, e.g. using Dropout or ensembles.
Propagation of uncertainty: estimate the reliability of the ranking scores, taking into account the uncertainty in the model’s output.

3. training time and resources:

Challenge: neural network-based ranking models require a lot of training time and computational resources due to the use of large datasets and complex architectures.
Solution:
Distributed training: use multiple GPUs and distributed computing to reduce training time.
Attempts to reduce weight: optimise resource use by reducing model complexity or using lightweight architectures.

4. overtraining:

Challenge: over-learning can occur with large models and large amounts of training data.
Solution:
Drop-out: add a drop-out layer to prevent over-learning.
Regularisation: use L1 regularisation or L2 regularisation to constrain model weights.
Ensemble: combine multiple models to improve generalisation performance.

5. appropriate choice of evaluation metrics:

Challenge: ranking is difficult to evaluate and the choice of appropriate evaluation metrics is important.
Solution:
Rank-related metrics: use metrics that adequately assess ranking performance, such as Normalised Discounted Cumulative Gain (NDCG), Mean Reciprocal Rank (MRR) and Mean Average Precision (MAP).
Indicators tailored to business objectives: tailor the evaluation indicators to the system’s objectives and business requirements.

6. interpretability and explainability:

Challenge: neural networks are black boxes, making results difficult to interpret and explain.
Solution:
Feature importance: calculate the feature importance of the model and understand its impact on ranking.
Local interpretation: use libraries such as SHAP and LIME to provide interpretation of individual predictions and rankings.

Reference Information and Reference Books

For general machine learning algorithms including search algorithms, see “Algorithms and Data Structures” or “General Machine Learning and Data Analysis.

“Algorithms” and other reference books are also available.

An Introduction to Neural Information Retrieval

1. Comprehensive Reviews and Surveys

2. Representative Model Papers

3. Recent Trends and Cutting-Edge Topics

4. Textbooks and Summary Books

コメント

タイトルとURLをコピーしました