An overview of maximum likelihood estimation and its algorithm and implementation

Machine Learning Artificial Intelligence Digital Transformation Deep Learning Information Geometric Approach to Data Mathematics Navigation of this blog

Maximum Likelihood Estimation

Maximum Likelihood Estimation (MLE) will be one of the estimation methods used in statistics. This method is used to estimate the parameters of a model based on given data or observations. Maximum likelihood estimation attempts to maximize the probability that data will be observed for different values of the parameters.

The basic idea of maximum likelihood estimation is to find the parameter values that maximize the probability of obtaining a given data, which is specifically done in the following steps.

Setting the likelihood function: The first step is to determine the probability distribution (likelihood function) that will produce the data. This distribution may depend on the parameters of the model.
Maximize the likelihood function: Viewing the likelihood function as a function of the parameters, we find the parameter values that maximize the likelihood for the given data. This is usually done by differentiating the likelihood function to find the maximum value.

Maximum likelihood estimation is widely used in many statistical models and machine learning algorithms. For example, maximum likelihood estimation is used in linear regression, logistic regression, and mixed normal distribution models. Maximum likelihood estimation has been theoretically shown to have properties of parameter unbiasedness and efficiency, and is commonly used as a statistical estimator.

However, maximum likelihood estimation has some limitations and caveats, especially the selection of initial values and numerical optimization methods because of the possibility of falling into local maxima. In addition, if the data contain noise or the model is not appropriate, the estimated parameters may also be inaccurate. Maximum likelihood estimation is a method to estimate parameters in a probabilistic framework, while Bayesian statistics uses Bayesian estimation as a method to estimate the posterior distribution.

Mathematical model of maximum likelihood estimation

To understand the mathematical model of maximum likelihood estimation, consider the following elements

Likelihood Function: The likelihood function is a function that indicates the probability that a given set of data will be generated with a particular parameter value. The likelihood function is defined as a function with the data fixed and the parameters as variables, and is usually expressed using a probability density function (or probability mass function) representing the probability distribution of the data.
Parameter estimation: The goal of maximum likelihood estimation is to find the values of the parameters that maximize the likelihood function. Mathematically, this can be accomplished by finding the maximum likelihood estimator by partial differentiation of the likelihood function by the parameters and equating the result to zero. This problem becomes an optimization problem, and it is common to use numerical optimization methods to find a solution.
Log-Likelihood Function: Likelihood functions are often very complex in form and can be difficult to compute. Therefore, the log-likelihood function (or log-likelihood) is often used. The log-likelihood function is a log transformation of the likelihood function, and the log transformation simplifies the computation.
Maximum Likelihood Estimate (MLE Estimate): The value of the parameter that maximizes the likelihood function is called the maximum likelihood estimator. The maximum likelihood estimator is the one that makes the most of the information available in the data to estimate the parameters and is considered a statistically good estimator.

The mathematical model of maximum likelihood estimation represents the relationship between the data and the model in a probabilistic framework and captures the process of estimating parameters from the data. Maximum likelihood estimation can be applied to a wide range of statistical problems and is used in a variety of fields.

Procedure and Algorithm for Maximum Likelihood Estimation

The procedure and algorithm for maximum likelihood estimation is as follows

Setting the likelihood function: The first step is to set the probability distribution (likelihood function) from which the data will be generated. This probability distribution depends on the parameters of the model, for example, the normal or Poisson distribution is often used.
Calculating the log-likelihood function: The log-likelihood function is a log-transformation of the likelihood function, which makes it easier to compute. Calculate the log-likelihood function.
Maximize the log-likelihood function: Find the parameter values that maximize the log-likelihood function. This is typically done using numerical optimization methods. Typical methods include gradient descent and Newton-Raphson methods. These methods are used to search for parameters such that the log-likelihood function is maximized.
Calculation of the maximum likelihood estimator: The parameter corresponding to the value of the maximized log-likelihood function is obtained as the maximum likelihood estimator.
Evaluating the results: Based on the estimated parameters, the goodness of fit of the model and the performance of the predictions are evaluated. This is important to confirm the validity of the estimation results.

The following is an example of an algorithm that conceptually illustrates the maximum likelihood estimation procedure.

Input: Dataset X, likelihood function f(θ|x), estimate of initial parameters θ₀, threshold for convergence condition ε
Output: Maximum likelihood estimator θ̂

1. θ = θ₀
2. Repeat:
   1. Calculate the value of the log-likelihood function:. L = ∑ log(f(θ|x)) for all x in X
   2. If the stop condition is met, the program will terminate:. if |L - L_previous| < ε, then break
   3. Parameter updates:.
      - Calculate gradient information:. ∂L/∂θ
      - Update Parameters:: θ = θ + learning_rate * ∂L/∂θ (Example: gradient descent method)
3. Output maximum likelihood estimator:. θ̂ = θ

The above algorithm estimates parameters through maximization of the log-likelihood function. Starting from the initial parameters, the parameters are updated in the direction of maximizing the log-likelihood function until a convergence condition is met, and the convergence condition is usually a small change in the log-likelihood function or when a pre-determined maximum number of iterations is reached.

Libraries and platforms used for maximum likelihood estimation

Maximum likelihood estimation is an important technique in statistics and machine learning and is supported by many programming languages and libraries. The following are some of the major libraries and platforms available for maximum likelihood estimation.

Python:
- NumPy: A numerical library that supports mathematical and data manipulation. It is used to implement and compute maximum likelihood estimation.
- SciPy: A library for scientific and technical computing, providing functions for maximum likelihood estimation, optimization, and statistical analysis.
- statsmodels: A library supporting statistical modeling, providing various statistical methods including maximum likelihood estimation.
R: a library of statistical modeling tools for the R programming language.
- stats: A standard statistical package for the R language, providing functions for maximum likelihood estimation and statistical modeling.
- lme4: A package that supports linear mixed-effects models and allows estimating parameters by maximum likelihood estimation.
Julia:
- Distributions.jl: Probability distribution processing library in the Julia language, providing functions for maximum likelihood estimation and manipulation of probability distributions.
MATLAB:
- MATLAB is a platform for numerical and scientific computing and also provides a toolbox for statistical modeling and maximum likelihood estimation.

Application of Maximum Likelihood Estimation

Maximum likelihood estimation is widely used to estimate parameters for various statistical models and machine learning algorithms. The following are some examples of its application.

Linear regression models: Linear regression models estimate regression coefficients that describe the relationship between explanatory and response variables. Maximum likelihood estimation can be used to find the regression coefficient that minimizes the error between the observed data and the predicted value.
Logistic Regression Model: The logistic regression model can be a classification algorithm used to predict two classes. Through maximum likelihood estimation, parameters (coefficients) are estimated to model the probability of belonging to each class.
Poisson regression model: used to model count data (e.g., the number of times an event occurs). The Poisson regression model estimates a parameter representing the average number of occurrences of an event through maximum likelihood estimation.
Mixed Normal Model: A mixed normal model is used when the data are generated from multiple normal distributions. The mean and variance of each component, as well as the mixture proportions, are obtained through maximum likelihood estimation.
Time Series Models: Used to model the behavior of data over time, maximum likelihood estimation is performed with time series models such as ARIMA models and state space models.
Non-negative matrix factorization: used to decompose a data matrix into multiple non-negative factor matrices. For example, maximum likelihood estimation is used in topic modeling and image processing.See “Overview of non-negative matrix factorisation (NMF) and examples of algorithms and implementations” in detail.
Structural Equation Modeling: Used to unravel relationships in data with complex model structures. Maximum likelihood estimation is used, for example, in path analysis and factor analysis.

Example implementation of estimating a linear regression model using maximum likelihood estimation

An example implementation using Python’s NumPy and SciPy libraries to estimate a linear regression model using maximum likelihood estimation is shown. In this example, data are generated appropriately and regression coefficients are estimated using maximum likelihood estimation.

import numpy as np
from scipy.optimize import minimize

# Data Generation
np.random.seed(0)
X = np.random.rand(50, 1)  # explanatory variable
y = 2 * X + 1 + 0.1 * np.random.randn(50, 1)  # Response variable (true regression coefficient is 2, intercept is 1)
# Likelihood function (mean squared error)
def likelihood(params):
    beta, intercept = params
    y_pred = X * beta + intercept
    mse = np.mean((y_pred - y) ** 2)
    return mse

# Initial Parameter Setting
initial_params = [1.0, 1.0]

# Perform maximum likelihood estimation
result = minimize(likelihood, initial_params, method='Nelder-Mead')

# Estimated Parameters
estimated_params = result.x
beta_estimated, intercept_estimated = estimated_params

print("Estimated regression coefficients:", beta_estimated)
print("Estimated intercept:", intercept_estimated)

The code generates data and performs maximum likelihood estimation using the likelihood function and optimization methods. The Nelder-Mead method is used as the optimization method, but other methods could be used (e.g., BFGS, L-BFGS-B, etc.).

Reference Information and Reference Books

For more information on optimization in machine learning, see also “Optimization for the First Time Reading Notes” “Sequential Optimization for Machine Learning” “Statistical Learning Theory” “Stochastic Optimization” etc.

Reference books include Optimization for Machine Learning

“Machine Learning, Optimization, and Data Science“

“Linear Algebra and Optimization for Machine Learning: A Textbook”