Introduction
Various models for emotion recognition have been proposed, as described in “Emotion recognition, Buddhist philosophy and AI“. In addition, a number of AI technologies such as speech recognition, image recognition, natural language processing and biometric analysis have been used to extract emotions. This section describes the details of these technologies.
Extracting emotions through speech recognition
It is possible to process speech data and analyse the speaker’s voice characteristics, speech rhythm and word choice to estimate the emotions of the speaker, and to combine speech recognition techniques with emotion recognition models to extract the speaker’s emotions.
Algorithms for recognising emotions in speech recognition mainly combine techniques such as speech signal processing, feature extraction, machine learning and deep learning. The following are common methods and approaches used to recognise emotions from speech.
1. speech signal processing: speech data is acquired as a waveform and pre-processed using signal processing techniques. This includes filtering, Fourier transforms, etc. Speech signal processing methods facilitate the extraction of useful information from speech data.
2. feature extraction: it is important to extract features from the speech data, with Mel Frequency Cepstrum Coefficient (MFCC), acoustic features, fundamental frequency (pitch) and energy being common features. These features hold information about the expression of emotion.
3. machine learning algorithms: various machine learning algorithms are used for emotion recognition. Support Vector Machines (SVMs), described in “Overview of Support Vector Machines, Applications and Various Implementations“, Decision Trees, described in “Overview of Decision Trees, Applications and Implementation Examples“, and Random Forests are common, and these algorithms classify emotions based on extracted features
4. deep learning algorithms: deep learning models are also used for emotion recognition. Recurrent Neural Networks (RNNs), described in “Overview of RNNs and Examples of Algorithms and Implementations“; Long Short-Term Memory Networks (LSTMs), described in “Overview of LSTMs and Examples of Algorithms and Implementations“; and GRUs, described in “Overview of GRUs and Examples of Algorithms and Implementations“. Models such as GRUs are better suited to modelling changes in emotion, as they are more likely to take into account temporal dependencies in the speech data.
5. Transfer learning: using a model that has been pre-trained for a general emotion classification task (e.g. a language model), the task of emotion recognition can be subjected to transfer learning as described in “Overview of transfer learning and examples of algorithms and implementations“. This allows for improved performance with less labelled data.
6. deep learning-based emotion recognition models: In emotion recognition, deep learning-based models are used to capture features of speech data. For example, Convolutional Neural Networks (CNNs), described in “CNN overview, algorithms and implementation examples“, are applied to speech spectrograms.
These algorithms are usually trained on large amounts of labelled data to acquire the ability to identify specific emotion categories. In emotion recognition, data collection and task-specific model selection are important, and the best algorithms for real-time processing and real-world use cases are being explored.
The procedure for emotion recognition with speech recognition is divided into several steps. The general steps are listed below.
1. data collection: collect labelled speech data, including emotions, to train speech recognition models. Data should be collected in a rich variety to cover different speakers, different contexts and different emotional states.
2. pre-processing: pre-process the collected speech data. This includes removing noise, normalising the speech, changing the sampling rate, etc., as well as sentence segmentation (detecting sentence breaks).
3. feature extraction from the speech signal: extracting features from the speech data. Typical features include Mel Frequency Cepstrum Coefficient (MFCC), fundamental frequency (pitch) and energy, which help to describe the nature of the speech data and information about emotions.
4. emotion labelling: emotion labels are assigned to the collected speech data. This indicates the emotional state that the speech expresses (e.g. joy, sadness, anger, etc.).
5. Splitting training and test data: Split the dataset into training and test data. Typically, most of the data is used for training and some for testing.
6. Model selection: select the emotion recognition model to be used. This includes machine learning algorithms (e.g. SVM, Random Forest), deep learning models (e.g. CNN, RNN, LSTM), etc.
7. train the model: train the selected model. Use training data to adjust weights so that the model can accurately recognise emotions from speech.
8. evaluating the model: using test data to evaluate the model’s performance. Check the performance of the model using metrics such as accuracy, recall and goodness of fit. 9.
9. tuning the model: if the model is not performing well, improve the model by adjusting hyper-parameters or adding new data.
10. prediction: once training and evaluation are complete, predict emotions for unknown speech data. The model estimates which emotional state the speech data corresponds to.
Through these steps, the emotion recognition model has acquired the ability to recognise emotions from speech data. Real-world applications require real-time processing and performance evaluation in different environments.
Several challenges exist in emotion recognition through speech recognition. The main challenges are described below.
1. lack and imbalance of data: emotion data sets can be insufficient, and furthermore, label imbalance can be a problem. In particular, if a particular emotion category does not have a sufficient number of samples compared to other categories, it becomes difficult for the model to recognise that emotion accurately.
2. diverse speakers: different speakers have different pronunciations, accents and speech styles, making it difficult for general emotion recognition models to cope with diverse speakers.
3. contextual understanding: it is important to understand the context of an utterance, as the same word or sound used in different contexts may express different emotions. If the model does not accurately capture the context, the performance of emotion recognition may be compromised.
4. phrasing diversity: the same emotion can be expressed in different words and phrases. This is referred to as phrase diversity, and the model needs to understand this and respond appropriately. 5. real-time processing requirements: the performance of models may suffer if they do not accurately capture the context.
5. real-time processing requirements: recognising emotions in real-time requires fast and efficient algorithms and models, especially in situations where real-time performance is required (e.g. voice assistants, customer service).
6. environmental noise: speech data is more likely to be degraded in noisy environments, and effective methods are required to cope with this noise.
7. protection of personal data: as speech data contains the speech of individuals, the protection of personal data is important and privacy needs to be considered when training and operating models.
Extracting emotions through image recognition and facial expression analysis
Emotions can also be extracted by analysing facial expressions and changes in facial expression. Facial recognition technology and deep learning models can be used to detect facial expressions and estimate emotions from photographs and videos.
Various algorithms and models based on machine learning and deep learning are used to analyse emotions using image recognition technology. The following are some of the most representative algorithms and models.
1. convolutional neural networks (CNNs): CNNs have been very successful in image classification tasks and are widely used for emotion analysis. It consists of a convolutional layer, a pooling layer and a total coupling layer to extract local features and recognise them in a hierarchical manner. For more information on CNNs, see “Overview of CNNs, Algorithms and Examples of Implementations“.
2. VGGNet: VGGNet is a model with a deep network structure of convolutional and pooling layers, and is widely used because of its simple and easy-to-understand structure. For more information, see About VGGNet.
3. ResNet (Residual Networks): ResNet provides a method for solving the gradient loss problem that occurs when building very deep networks. This facilitates the construction of deep networks and performs well in sentiment analysis. For more information, see About ResNet (Residual Network).
4. Inception (GoogLeNet): Inception provides a network structure that can capture features at multiple scales by applying filters of different sizes simultaneously. For more information on Inception, see “About GoogLeNet (Inception)“.
5. MobileNet: MobileNet is a lightweight and efficient network structure suitable for running on mobile devices. For more information on MobileNet, see “About MobileNet“.
6. Xception: Xception is based on the Inception model, but adopts an approach where convolutional operations are applied independently in a deep network structure to improve computational efficiency while maintaining high expressive power.
These models are typically pre-trained on large datasets and applied to the task of sentiment analysis through transfer learning and fine tuning. In addition, datasets may be used in emotion analysis where these models are devised to take into account not only facial features, but also pose and environment.
The following section describes the general procedure and techniques used for emotion analysis using image recognition techniques.
1. data collection: for emotion analysis, a labelled emotion image dataset is required, which contains images of people in various emotional states.
2. face detection: face detection techniques are used to detect faces in images. This identifies the areas to be analysed for emotion analysis. Common methods include Haar Cascades and deep learning-based face detection models (e.g. MTCNN, Dlib), as described in “Overview of Haar Cascades and examples of algorithms and implementations“.
3. face feature extraction: Once the face is detected, the facial features are extracted. This includes feature points for each part of the face and features to represent facial expressions, in particular the position of the eyes, the movement of the eyebrows and the shape of the mouth to capture facial expressions.
4. emotion classification model training: the extracted features are used to train the emotion classification model. This can be done using machine learning algorithms or deep learning models.
5. Emotion prediction: once training is complete, emotion prediction is performed on the unknown image. The model outputs the emotions in the image as classes (e.g. joy, anger, surprise, etc.).
Some of the main challenges and caveats of analysing emotions using image recognition techniques include.
- Data diversity: if the training data is not diverse, the model may only be valid under certain conditions. It is important to train models using diverse data sets.
- Face occlusion or positional changes: if a face is occluded by another object or hand, or if the position of the face in the image changes, the accuracy of the emotion analysis may be reduced.
- Environmental influences: the model is susceptible to light conditions and background influences and must be robust to these environmental conditions.
Emotion extraction using natural language processing
Natural language processing (NLP) emotion extraction can be a technique for detecting emotions and emotional states from textual data. The following methods and techniques are commonly used for this:
1. rule-based approaches:
– Dictionary-based methods: a dictionary of emotion-related words and phrases is created in advance and matched with words in the text. For example, ‘happy’ and ‘joyful’ are considered positive emotions, while ‘sad’ and ‘angry’ are considered negative emotions.
– Rule-based systems: extract emotions based on specific grammar rules or patterns. For example, if the phrase ‘I feel [emotion]’ appears, it would extract that emotion.
2. machine learning-based approaches:
– Classification models: classify text into emotion categories using traditional machine learning algorithms such as support vector machines (SVMs), random forests and naïve Bayes, and train models using large amounts of labelled data.
– Neural networks: recurrent neural networks (RNNs) and long short-term memory (LSTM) networks are particularly commonly used, as they are well suited to capture contextual information in text, allowing for highly accurate sentiment classification.
3. deep learning-based approaches:
– Transformer models such as BERT and GPT: these models are based on large pre-trained language models and perform very well for sentiment extraction. These models take into account the context and analyse the text to estimate emotions.
4. hybrid approaches:
– This approach combines rule-based and machine learning-based methods to improve the accuracy of emotion extraction. For example, a basic emotion dictionary can be used for initial classification and fine-tuned with machine learning models.
Examples of applications include.
– Customer feedback analysis: analysing customer reviews and feedback to understand sentiment towards a product or service.
– Social media monitoring: analysing social media posts to understand public opinion on a brand or topic.
– Customer support: analyse chat and email content to understand customers’ emotional state and respond appropriately.
– Emotional interfaces: improve user experience by adjusting interfaces and responses according to emotional states.
The collection and pre-processing of appropriate data is crucial for successful emotion extraction, in particular, the quality and quantity of labelled data has a significant impact on the performance of the model. Attention should also be paid to multilingual support and cultural background differences.
Emotion extraction through biometric analysis
Emotion recognition using body language information, including brain observation as described in “Brain-machine interface applications and OpenBCI“, has been approached in various ways as an application of IOT technology as described in “Sensor data & IOT technology“.
Emotion recognition by analysing biometric information mainly uses the following biometric signals to infer an individual’s emotional state:
1. heart rate (HR): fluctuations in heart rate are associated with emotional states such as stress and excitement. A high heart rate indicates tension or stress, while a low heart rate indicates a relaxed state.
2. heart rate variability (HRV): Heart rate variability refers to the variation in the time interval between heartbeats and reflects stress levels and states of relaxation. High HRV indicates a relaxed state, while low HRV indicates stress or fatigue. 3.
3. skin electrical activity (EDA): skin conductivity is sensitive to emotional responses and changes during stress and excitement. Increased sweat gland activity in the skin also increases skin electrical activity.
4. brain waves (EEG): EEG measures the electrical activity of the brain and certain frequency bands are associated with different emotional states. For example, alpha waves indicate a state of relaxation or meditation, while beta waves indicate a state of concentration or stress.
5. breathing patterns: the rhythm and depth of breathing also reflect emotional states. Fast, shallow breathing indicates stress and anxiety, while slow, deep breathing indicates relaxation.
6. facial expression recognition: the technique of recognising emotions from facial expressions is also common, analysing facial muscle movements and features to identify emotions such as joy, sadness, surprise, anger, etc. 7. voice analysis: voice analysis can be used to identify the sound of a person’s voice.
7. speech analysis: speech features such as tone, pitch and rhythm are analysed to infer emotional states, with changes in voice reflecting emotions such as tension, excitement and anger.
These biometric signals can not only be analysed individually, but also combined with multiple signals for more accurate emotion recognition. For example, more detailed emotional states can be ascertained by analysing heart rate, skin electrical activity and brain waves simultaneously.
Applications of biometric-based emotion recognition include mental health care, improved user experience and improved human-computer interaction.
Reference Information and Reference Books
For more information on voice recognition technology, please refer to “Speech Recognition Technology.
Reference book were “Automatic Speech Recognition: A Deep Learning Approach“
“Robust Automatic Speech Recognition: A Bridge to Practical Applications“
“Audio Processing and Speech Recognition: Concepts, Techniques and Research Overviews“
コメント