Overview of parallel distributed processing in machine learning and examples of on-prem/cloud implementations

Machine Learning Artificial Intelligence Digital Transformation ICT Technology Sensor Data & IOT ICT Infrastructure Stream Data Processing Parallel & Distributed Processing Navigation of this blog

Parallel and Distributed Processing in Machine Learning

Parallel and distributed processing in machine learning distributes data and computations across multiple processing units (CPUs, GPUs, computer clusters, etc.) and simultaneously processes them to reduce processing time and improve scalability, which is important when processing large data sets and complex models It plays an important role in processing large data sets and complex models. The following sections describe these common methods and their implementations in machine learning.

Data Parallel Processing: Data parallel processing is a technique in which data is divided into multiple processing units, each of which processes the data independently, and each processing unit acquires a partial data set and performs model training and prediction in parallel. The final result is obtained by aggregating the processing results of the individual processing units. For example, an example implementation of data parallel processing using the TensorFlow library is shown below.

import tensorflow as tf
import numpy as np

# Data Preparation
X = ...  # feature vector
y = ...  # target label

# Split data and distribute to each processing unit
X_split = np.array_split(X, num_workers)  # num_workers is the number of processing units
y_split = np.array_split(y, num_workers)

# Perform learning in each processing unit
model = tf.keras.Sequential([...])  # Model Definition
optimizer = tf.keras.optimizers.SGD(...)  # Optimizer Definition
for i in range(num_workers):
    with tf.device(f'/cpu:{i}'):  # Designation of processing units
        model.compile(optimizer=optimizer, ...)
        model.fit(X_split[i], y_split[i], ...)

# Aggregation of processing results
aggregated_model = tf.keras.Sequential([...])
aggregated_model.set_weights(model.get_weights())

The above example shows a flow in which the data is divided into multiple processing units and training is performed on each unit. Each unit builds the model independently, performs training on a partial data set, and finally, aggregates the weights of each unit to obtain the final model.

Model Parallel Processing: Model parallel processing is a method of dividing a complex model into multiple processing units, each of which processes a portion of the model. Each processing unit is responsible for a partial model and shares the processing results with other units while learning and predicting the model. Below is an example implementation of model parallel processing using the PyTorch library.

import torch
import torch.nn as nn
import torch.optim as optim

# Data Preparation
X = ...  # feature vector
y = ...  # target label

# Model partitioning and placement
model_part1 = nn.Sequential([...])  # Part of model
model_part2 = nn.Sequential([...])  # Other parts of the model
model_part1.to('cuda:0')
model_part2.to('cuda:1')

# Placement of split data
X_part1 = X[:, :100]  # Part of data
X_part2 = X[:, 100:]  # Other parts of the data
y_part1 = y  # Labels

# Model Learning and Prediction
criterion = nn.CrossEntropyLoss()
optimizer1 = optim.SGD(model_part1.parameters(), lr=0.001)
optimizer2 = optim.SGD(model_part2.parameters(), lr=0.001)
for epoch in range(num_epochs):
    output_part1 = model_part1(X_part1)
    output_part2 = model_part2(X_part2)
    loss1 = criterion(output_part1, y_part1)
    loss2 = criterion(output_part2, y_part1)
    optimizer1.zero_grad()
    optimizer2.zero_grad()
    loss1.backward()
    loss2.backward()
    optimizer1.step()
    optimizer2.step()

In the above example, the model is split into two processing units, with each unit processing a portion of the data. Part of the model is placed in cuda:0 and the other part in cuda:1, training is done in each, and the final prediction is obtained by combining the output of both units.

Parallel and distributed processing in machine learning plays an important role in improving processing efficiency, handling large-scale data, and training complex models, and since each framework and library has functions and tools that support parallel and distributed processing, it is recommended that they be used in implementation.

Libraries and platforms used for parallel and distributed processing in machine learning

Various libraries and platforms are used to achieve parallel and distributed processing in machine learning. Some of the most representative ones are described below.

TensorFlow: TensorFlow is an open source machine learning framework developed by Google that supports parallel and distributed processing. TensorFlow provides distributed strategies and APIs for data and model parallelism, facilitating parallel processing on multiple devices and machines.
PyTorch: PyTorch is a machine learning framework developed by Facebook that supports parallel and distributed processing. pyTorch provides a number of parallelization strategies, including torch.nn.DataParallel and torch.nn.parallel. DistributedDataParallel and other functions can be used to implement data parallelism and model parallelism.
Horovod: Horovod is a distributed deep learning framework developed by Uber that can be integrated with other frameworks such as TensorFlow and PyTorch. kits to facilitate data parallelism and model parallelism, allowing for efficient training on multiple machines.
Apache Spark: Apache Spark is a distributed processing platform for large-scale data processing that also supports machine learning. Spark provides specialized APIs for parallel processing of data and distributed training of models to support the development of highly scalable machine learning applications.

Next, we will discuss some examples of applications of these parallel and distributed processes.

On the Application of Parallel and Distributed Processing in Machine Learning

Parallel and distributed processing in machine learning has been effectively used in the following applications

Big data processing: Parallel and distributed processing is very important when processing large amounts of data. Using data parallel processing, data can be divided into multiple nodes and processed in parallel to improve the efficiency of data processing. For example, the distributed data processing framework Apache Spark can be used to pre-process large data sets and extract features in parallel. Model Training.
Model Training: Training complex machine learning models can be computationally intensive. Using model parallel processing, models can be split into multiple nodes and trained in parallel, significantly reducing training time. Methods such as hyperparameter tuning may also be performed using parallel distributed processing.
Real-time Prediction: Processing speed is critical for real-time prediction and inference. By distributing model parameters across multiple nodes and performing forecasting in parallel, high-speed real-time processing can be achieved. This is an important factor in applications such as real-time personalization of online advertising and automated driving.
Hyperparameter Search: Hyperparameter search of models is important for improving machine learning performance, but the computational cost can be very high. Parallel distributed processing can be applied to this problem by simultaneously evaluating different hyperparameter combinations on multiple nodes to search for the optimal combination.

These are only a few examples of applications, and parallel distributed processing can be effective in a wide variety of machine learning tasks and problems. For specific applications, it is important to select the appropriate parallel and distributed processing method depending on the problem, such as parallel processing of data, parallel processing of models, real-time processing, and hyper-parameter search.

Next, we will discuss a case using Apache Spark as a concrete implementation example.

About setting up an Apache Spark environment

There are several steps to setting up and getting up and running with Apache Spark, as described in “Introducing and Setting Up Apache Spark for Distributed Data Processing,” but this section describes the simplest steps.

Install Java: Since Apache Spark runs on Java, you must first install Java by downloading and installing the Java Development Kit (JDK) from Oracle’s official website.
Download Spark: Download the version of Spark you wish to use from the official Apache Spark website; Spark is pre-built and comes as a single binary file.
Extract Spark: Extract the downloaded Spark file to an appropriate directory. The extracted directory will be the home directory of Spark.
Environment variable settings: To run Spark, environment variables must be set: Create an environment variable named SPARK_HOME and specify the path to the Spark home directory. Also, add the path to the Spark binary directory to the PATH environment variable.
Edit the configuration file: Edit the Spark configuration file to make the necessary settings: edit the spark-defaults.conf file in the SPARK_HOME/conf directory to specify the cluster settings, log storage location, etc.
Start Spark: To start Spark, use the spark-submit command from the command line. spark-submit command allows you to specify the path of the application to run and any necessary arguments, as well as settings such as master URL and deployment mode.

The following section describes an example implementation of distributed processing using these commands.

For example implementations of Apache Spark

Apache Spark is a distributed processing platform for large-scale data processing that also supports machine learning. The following sections describe examples of machine learning implementations using Apache Spark.

Data Import and Preprocessing: Apache Spark provides the ability to import data from a variety of data sources. The following is an example of reading and preprocessing data in CSV format.

from pyspark.sql import SparkSession
from pyspark.ml.feature import StringIndexer, VectorAssembler
from pyspark.ml import Pipeline

# Creating a Spark Session
spark = SparkSession.builder.getOrCreate()

# Loading Data
data = spark.read.csv('data.csv', header=True, inferSchema=True)

# Establishment of pre-processing pipeline
string_indexer = StringIndexer(inputCol='label', outputCol='label_index')
assembler = VectorAssembler(inputCols=['feature1', 'feature2'], outputCol='features')
pipeline = Pipeline(stages=[string_indexer, assembler])

# data conversion
transformed_data = pipeline.fit(data).transform(data)

In the above example, data is read from a CSV file, and then a preprocessing pipeline is built to transform the data by encoding labels and combining feature vectors.

Learning and Evaluating Machine Learning Models: Apache Spark offers a variety of machine learning algorithms. The following is an example of training and evaluating a logistic regression model for a classification task.

from pyspark.ml.classification import LogisticRegression
from pyspark.ml.evaluation import BinaryClassificationEvaluator
from pyspark.ml.tuning import ParamGridBuilder, CrossValidator

# Split model into training and test data
train_data, test_data = transformed_data.randomSplit([0.7, 0.3], seed=42)

# Model Learning
lr = LogisticRegression(featuresCol='features', labelCol='label_index')
param_grid = ParamGridBuilder().addGrid(lr.regParam, [0.01, 0.1]).build()
evaluator = BinaryClassificationEvaluator(metricName='areaUnderROC')
cv = CrossValidator(estimator=lr, estimatorParamMaps=param_grid, evaluator=evaluator, numFolds=5)
cv_model = cv.fit(train_data)

# Model Evaluation
predictions = cv_model.transform(test_data)
auc = evaluator.evaluate(predictions)

In the above example, the data is split into training and test data, and the logistic regression model is trained. Hyperparameter tuning is done using cross-validation to evaluate the prediction results on the test data.

Performing Distributed Processing: Apache Spark also provides the ability to work with cluster managers and distributed file systems to easily perform distributed processing. Below we describe an example of running distributed processing on a Spark cluster.

from pyspark import SparkConf
from pyspark import SparkContext

# Spark Configuration Settings
conf = SparkConf().setAppName('MyApp').setMaster('spark://localhost:7077')
sc = SparkContext(conf=conf)

# Creation and operation of RDDs
rdd = sc.parallelize([1, 2, 3, 4, 5])
rdd_squared = rdd.map(lambda x: x ** 2)
result = rdd_squared.collect()

In the above example, the Spark config is set and the Spark context is created. The RDD is then created and the map operation is applied to square each element. The final results are collected using the collect() method.

Apache Spark becomes a tool that supports a variety of machine learning tasks, from data preprocessing to training and evaluating models to performing distributed processing.

Next, we discuss parallel and distributed processing in a cloud environment (AWS).

Machine Learning with Parallel and Distributed Processing on AWS

While the simple use of AWS (Amazon Web Services) for parallel distributed processing is to build the above-mentioned spark on a container, here we use a different approach: we describe the execution of a machine learning workload by parallel distributed processing using AWS. This section describes the following.

The following services and approaches are available for machine learning with parallel distributed processing in AWS.

Amazon Elastic MapReduce(EMR)：

EMR is a managed service for distributed processing of large data sets using open source frameworks such as Hadoop and Spark, etc. EMR allows you to create clusters and run tasks on multiple nodes in parallel. The following outlines the steps for setting up an environment using Amazon Elastic MapReduce (EMR) and implementing with it.

1. Sign in to the AWS Management Console.
2. Go to the EMR console.
3. Click on “Create Cluster.
4. In the “Basic Settings” section, configure basic information such as the cluster name and log folder location.
5. In the Software Settings section, select the Hadoop version you want to use and any additional software packages (Spark, Presto, etc.).
6. In the Hardware Settings section, specify the number of master and core nodes, instance type, etc.
7. In the Security Settings section, configure security-related settings such as EC2 key pairs and security groups.
8. In the “Startup Settings” section, specify the scripts and job flows to be executed at startup.
9. In the “Steps” section, configure the details of the job to be executed. This allows you to specify Spark and Hive jobs, for example.
10. In the “Bootstrap Actions” section, you can specify additional scripts to be executed when the cluster is started.
11. After completing the necessary settings, click “Create Cluster” to create the EMR cluster.

The EMR cluster is now created and ready to perform machine learning tasks.

As an implementation example, we will discuss a case where data preprocessing and model training are performed using Spark.

1. Create an EMR cluster.
2. Once the cluster is started, configure SSH access to the EMR master node.
3. Connect to the master node via SSH and create Spark code. Here, for example, the following code can be created using PySpark.

from pyspark import SparkContext, SparkConf

conf = SparkConf().setAppName("MySparkJob")
sc = SparkContext(conf=conf)

# Write data loading and preprocessing code

# Write code for model training and inference

sc.stop()

1. Create a script file (e.g. my_spark_job.py) to run the code.
2. Return to the master node via SSH and execute the following command to run the Spark job

spark-submit my_spark_job.py

This will run Spark on the EMR cluster and execute the specified code in parallel. It is important to note that while EMR can be used as a temporary cluster, it must be properly designed and managed for use in a production environment, and since the actual implementation involves various factors such as data loading, storage, and model management, the implementation Therefore, adjustments to the implementation should also be considered.

AWS Batch：

AWS Batch will be a service for scheduling and executing batch processing workloads. Tasks such as machine learning model training and inference can be distributed across multiple computing resources for parallel execution. Below is a step-by-step example of a machine learning implementation using AWS Batch.

1. Sign in to the AWS Management Console.
2. Navigate to the AWS Batch console.
3. Click on “Create Job Queue” to create a job queue. A job queue is a queue that manages the execution of jobs.
4. Click “Create Job Definition” to create a job definition. A job definition defines the execution method and parameters of the job.
5. In the job definition, set the computing environment to be used; AWS Batch will run the job on an EC2 instance.
6. The job definition specifies the container image to run. This image will contain the machine learning framework, required packages, model training scripts, etc.
7. In the job definition, specify the commands and arguments to be executed. This will be the paths and arguments of the model training scripts and inference scripts.
8. In the job definition, set the required resources (CPU, memory).
9. Once the job definition is created, click “Create Job” to create the job. A job is a task that will be executed using the job definition.
10. When the job is created, job parameters and input data location are specified.
11. Once the job is created, AWS Batch schedules and executes the job in the specified job queue.

This allows AWS Batch to execute multiple jobs in parallel and distribute the machine learning task. As an example implementation, consider the following steps

1. Go to the AWS Batch console and click on “Create Job Queue”.
2. Create a job queue and configure job priorities, resource allocation policies, etc.
3. Click “Create Job Definition” to create a job definition.
4. In the job definition, specify the Docker container image to be used for execution. For example, use an image containing a machine learning framework such as TensorFlow or PyTorch.
5. In the job definition, specify the scripts, commands, and arguments to be executed. For example, specify the path and arguments for a model training script.
6. In the job definition, set the required resources (CPU, memory).
7. Click “Create Job” to create a job.
8. When creating a job, specify the job parameters and location of input data.
9. Once the job is created, AWS Batch schedules and executes the job in the specified job queue.

AWS Step Functions：

Step Functions is a service for creating and executing workflows, which can help build a machine learning pipeline. It allows you to define multiple steps, execute them in parallel, and control the flow of processing based on conditions. Below are step-by-step instructions for an example machine learning implementation using AWS Step Functions.

1. Sign in to the AWS Management Console.
2. Navigate to the AWS Step Functions console.
3. Click Create State Machine.
4. Create a state machine definition. The definition is a JSON-formatted document that describes the order of tasks, branches, parallel processing, and so on.
5. In the state machine definition, describe the machine learning tasks. For example, define the tasks of data preprocessing, model training, and inference.
6. To execute tasks in parallel, use the “Parallel” state of Step Functions. This allows multiple tasks to be executed simultaneously.
7. Each task can specify an execution resource, such as an AWS Lambda function or an AWS Glue job. It also allows for integration with different AWS services and external APIs.
8. Once the state machine definition is complete, click “Create State Machine” to create the Step Functions state machine.
9. Once the state machine is created, start the state machine by specifying the input data.

This will allow AWS Step Functions to execute the defined workflow and perform machine learning tasks in parallel distributed processing. As an example implementation, consider the following steps

1. Go to the AWS Step Functions console and click on “Create State Machine”.
2. In the state machine definition, describe the sequential and parallel states. For example, consider the following definition

{
  "StartAt": "DataPreprocessing",
  "States": {
    "DataPreprocessing": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:DataPreprocessingFunction",
      "End": true
    },
    "ModelTraining": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:ModelTrainingFunction",
      "End": true
    },
    "Inference": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:InferenceFunction",
      "End": true
    },
    "ParallelTasks": {
      "Type": "Parallel",
      "Branches": [
        {
          "StartAt": "DataPreprocessing",
          "States": {
            "DataPreprocessing": {
              "Type": "Task",
              "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:DataPreprocessingFunction",
              "End": true
            }
          }
        },
        {
          "StartAt": "ModelTraining",
          "States": {
            "ModelTraining": {
              "Type": "Task",
              "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:ModelTrainingFunction",
              "End": true
            }
          }
        },
        {
          "StartAt": "Inference",
          "States": {
            "Inference": {
              "Type": "Task",
              "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:InferenceFunction",
              "End": true
            }
          }
        }
      ],
      "End": true
    }
  }
}

1. Specify the ARN (Amazon Resource Name) of the Lambda function to be executed as the Resource for each state.
2. Click Create State Machine to create an AWS Step Functions state machine.
3. Once the state machine is created, specify the input data to start the state machine.

AWS Step Functions can execute tasks in parallel based on state machine definitions, monitor the completion of each task, and flexibly define control logic such as dependencies, branches, and loops between tasks.

AWS Glue：

Glue will be a fully managed ETL (Extract, Transform, Load) service for data preparation and transformation. This will allow for parallel pre-processing and feature engineering of large data sets. Below is an overview of the steps to implement a machine learning workload using AWS Glue.

1. Sign in to the AWS Management Console.
2. Navigate to the AWS Glue console.
3. Click on “Create Job.”
4. In the “Job Settings” section, specify the name, role, and job type (Script or Apache Spark) for the job.
5. In the “Data Source” section, specify the source of the input data (S3 bucket, RDBMS, data warehouse, etc.).
6. In the Target section, specify the target of the output data (S3 bucket, RDBMS, data warehouse, etc.).
7. In the “Job Script” section, describe the job script. Job scripts describe machine learning tasks such as input data transformation and cleaning, feature engineering, and model training.
8. In the “Job Execution Settings” section, configure settings related to job execution. For example, resource allocation, job scheduling, and triggers can be specified.
9. After completing the necessary settings, click “Create Job” to create an AWS Glue job.

The AWS Glue job is now created, and the machine learning task is ready to be executed in parallel distributed processing. Next, as an example implementation, consider the following steps.

1. Go to the AWS Glue console and click on “Create Job”.
2. In the Job Settings, specify the name, role, and job type for the job.
3. Specify data in an S3 bucket as the data source.
4. Specify another S3 bucket as the target to store the model training and prediction results.
5. Create a job script to describe the machine learning task. For example, write code to preprocess the data and train the model using Python’s pandas or scikit-learn libraries.
6. Configure resource allocation and job scheduling in the job execution settings.
7. Click Create Job to create an AWS Glue job.

AWS Glue automatically performs parallel distributed processing even if the specified data source or target is large. Glue also uses Apache Spark as its backend, providing fast processing and scalability.

Reference Information and Reference Books

More detailed information on machine learning with parallel distributed processing is given in “Parallel Distributed Processing in Machine Learning“. Please refer to that as well.

Reference books also include”Machine Learning Engineering on AWS: Build, scale, and secure machine learning systems and MLOps pipelines in production”

“Parallel and Distributed Computing, Applications and Technologies“

“Parallel Distributed Processing: Explorations in the Microstructure of Cognition“