Speech Emotion Detection using Machine Learning

Speech emotion detection is the expanded form of (SED) and it is a synthesizing field that takes place at the crossroads of speech processing and emotion recognition. It mainly focuses on analysing the emotional state of a speaker depending on their vocal features.

The common methods for developing an SED (Speech emotion detection) project is described here,

1. Problem Definition :

• Deriving out of the provided speech instance, we detect and categorize the emotional condition of a speaker.

2. Data Collection :

• We require the classified audio data, where every individual sample is connected with specific emotion. The database such as RAVDESS, EmoReact, and CREMA-D consists of speech samples defined with emotions.

3. Data Pre-processing :

• Segmentation: In case we possess long audio clips, then divide them into smaller and fixed-size portions.
• Sampling Rate: Make sure that audio clips are at a logical specimen rate (e.g., 16 kHz).
• Silence Removal: Separate the silent portions of the audio to decrease the meaningless information.

4. Feature Extraction :

     This is one of the incredibly significant methods. The following are the some of the common features we use in SED,

• Mel-Frequency Cepstral Coefficients (MFCCs): These coefficients are conjointly inventing the MFC.
• Chroma-STFT: It is associated with twelve different pitch classes.
• Mel-Scaled Spectrogram: This measures the pitches which are examined by listeners must be similar in space one from another.
• Contrast: In the frequency spectrum, the dissimilarities between peak and trough are mentioned.
• Tonnetz: It estimates the tonal centroid features.

5. Exploratory Data Analysis (EDA) :

• We figure out the allocation of numerous emotions in the dataset.
• Utilize tools like ,
• Librosa for visualizing the spectrograms
• Wave plots
• Mel-Frequency Cepstral Coefficients (MFCCs) of audio samples.

6. Model Selection :

• Classical Machine Learning: The algorithm applied by us like Support Vector Machine (SVM), Random Forest or Gradient Boosting Machines are the best start up points.
• Deep Learning: Deploy Convolutional Neural Networks (CNNs) on spectrogram images and Recurrent Neural Networks (RNNs) or Long-Short Term Memory systems (LSTMs) are occupied on sequences of derived attributes.

7. Model Training :

• Distribute our model data into training, validation and test sets.
• On the training dataset, the model gets trained and validates it on the dataset of validation.

8. Evaluation :

• Accuracy: This is beneficial for balanced datasets.
• Precision, Recall, F1-score: If we suppose to face class imbalance, then employ the significant metrics.
• Confusion Matrix: It contributes better understanding within the particular miscalculation among emotions.

9. Optimization :

• The model hyper parameters are altered and progressed by us.
• Test the model with varied architectures or feature sets.

10. Deployment :

• Our model is combined within an application that claims real-time emotion detection. Such areas like call centres, voice assistants and therapy apps.

11. Feedback Loop :

• The users are allowed to submit their reviews on emotion detection. Moreover, it is helpful for us to make improvements and retrain the model.

Techniques and References:

• Data Handling & Feature Extraction: The tools involved are librosa and scipy.
• Modelling: TensorFlow, Keras, PyTorch and scikit-learn are the methods used in modelling.
• Visualization: We occupy visualization tools like Matplotlib and Seaborn.

Terminal Ideas:

Speech Emotion Detection (SED) is playing a major role in numerous areas like customer service, entertainment and healthcare. At the same time, it is crucial to manage the applications properly and make sure users are knowledgeable about their emotions are being identified and checking data privacy.

1. Speech emotion recognition and classification using hybrid deep CNN and BiLSTM model

Abstract

            Accurate emotion detection from speech utterances has been a challenging and active research affair recently. Speech emotion recognition (SER) systems play an essential role in Human-machine interaction, virtual reality, emergency services, and many other real-time systems. It is an open-ended problem as subjects from different regions and lingual backgrounds convey emotions altogether differently. The conventional approach used low-level periodic features from audio samples like energy, pitch, etc., for classification but was not efficient enough to detect emotions accurately and not generalized. With the recent advancements in computer vision and neural networks extracting high-level features and more accurate recognition can be achieved. This study proposes an ensemble deep CNN + Bi-LSTM-based framework for speech emotion recognition and classification of seven different emotions. The paralinguistic log Mel-frequency spectral coefficients (MFSC) is used as a feature to train the proposed architecture. The proposed Hybrid model is validated with TESS and SAVEE datasets. Experimental results have indicated a classification accuracy of 96.36%. The proposed model is compared with existing models, proving the superiority of the proposed hybrid deep CNN and Bi-LSTM model.

Keywords

Speech emotion recognition, Deep convolutional neural networks, LSTM, MFSC, Ensemble learning

            An integrated Deep CNN and Bi-LSTM related system for speech emotion detection and categorization of various emotions is suggested in our paper. To train the recommended system, MFSC is utilized as a feature.  By utilizing different datasets, our suggested integrated system is evaluated. As a consequence, we compared our model with other previous techniques and results show that, this integrated system outperforms other methods.

2. SCQT-MaxViT: Speech Emotion Recognition with Constant-Q Transform and Multi-Axis Vision Transformer

Abstract

            Speech emotion recognition presents a significant challenge within the field of affective computing, requiring the analysis and detection of emotions conveyed through speech signals. However, existing approaches often rely on traditional signal processing techniques and handcrafted features, which may not effectively capture the nuanced aspects of emotional expression. In this paper, an approach named “SCQT-MaxViT” is proposed for speech emotion recognition, combining signal processing, computer vision, and deep learning techniques. The method utilizes the Constant-Q Transform (CQT) to convert speech waveforms into spectrograms, providing high-frequency resolution and enabling the model to capture intricate emotional details. Additionally, the Multi-axis Vision Transformer (MaxViT) is employed for further representation learning and classification of the CQT spectrograms. MaxViT incorporates a multi-axis self-attention mechanism, facilitating both local and global interactions within the network and enhancing the ability of the model to learn meaningful features. Furthermore, the dataset is augmented using random time masking techniques to enhance the generalization capabilities. Achieving accuracies of 88.68% on the Emo-DB dataset, 77.54% on the RAVDESS dataset, and 62.49% on the IEMOCAP dataset, the proposed SCQT-MaxViT method exhibits promising performance in capturing and recognizing emotions in speech signals.

Keywords

Spectrogram, Constant-Q transform, Vision transformer, Multi-axis vision transformer, Emo-DB, RAVDESS, IEMOCAP.

               A technique called SCQT-MaxViT is recommended in our system for speech emotion recognition integrating computer vision, DL approaches and signal processing. By using Constant-Q Transform, speech waveforms are transformed into spectrograms that give high frequency resolution and allowing the system to retrieve important emotional information. To categorize the CQT spectrograms, MaxViT is utilized in our approach.

3. Speech emotion recognition using multimodal feature fusion with machine learning approach

Abstract

            Speech-based emotional state recognition must have a significant impact on artificial intelligence as machine learning advances. When it comes to emotion recognition, proper feature selection is critical. As a result, feature fusion technology is offered in this work as a means of achieving high prediction accuracy by emphasizing the extraction of sole features. Mel Frequency Cepstral Coefficients (MFCC), Zero Crossing Rate (ZCR), Mel Spectrogram, Short-time Fourier transform (STFT) and Root Mean Square (RMS) are extracted, and four different feature fusion techniques are used on five standard machine learning classifiers: XGBoost, Support Vector Machine (SVM), Random Forest, Decision-Tree (D-Tree), and K Nearest Neighbor (KNN). The successful use of feature fusion techniques on our suggested classifier yields a satisfactory recognition rate of 99.64% on the female only dataset (TESS), 91% on SAVEE (male only dataset) and 86% on CREMA-D (both male and female) dataset. The proposed model shows that effective feature fusion improves the accuracy and applicability of emotion detection systems.

Keywords

Feature fusion (FF), Zero Crossing Rate, Support vector machine, XGBoost

            In our approach, feature fusion method is employed by focusing the extraction of single features for accurate emotion recognition model. MFCC, ZCR, Mel Spectrogram, Short-time Fourier transform (STFT) and Root Mean Square (RMS) are extracted.  Various feature fusion methods are implemented on several ML approaches like XGBoost, SVM, Random Forest, D-Tree, and KNN.

4. A Review on Emotion Based Harmful Speech Detection Using Machine Learning

Abstract

            The paper represents the state-of-the-art review of the machine learning methods for hate speech detection. This paper reviews novel applications of machine learning algorithms in hate speech. The machine learning based three algorithms i.e., Long-Short Term Memory, random forest, convolution neural network found to be most useful in hate speech detection. These algorithms are found to be most useful for twitter, Facebook, and other social platforms. This paper briefly surveys the most usable deep learning algorithms for detecting the hate speech in Arabic, English, Hindi, and other languages. The review result shows that the mentioned machine learning algorithms give an excellent result over other deep learning algorithm. Therefore, these three algorithms are widely acceptable for the evaluation of hate speech.

Keywords

Hate Speech, Machine Learning, Deep Learning, Random Forest

            Various existing novel applications of ML techniques are reviewed in our article for the detection of hate speech. We stated Several ML based techniques like Long-Short Term Memory, random forest and convolution neural network are suitable for detection of hate speech. This article surveyed the DL techniques that are suitable for detecting hate speeches in several of languages in Arabic, English, Hindi and others.

5. Speech Emotion Recognition using Machine Learning

Abstract

            The aim of the paper is to detect the emotions which are elicited by the speaker while speaking. Emotion Detection has become an essential task these days. The speech which is in fear, anger, joy have higher and wider range in pitch whereas have low range in pitch. Detection of speech is useful in assisting human machine interactions. Here we are using different classification algorithms to recognize the emotions , Support Vector Machine , Multi-layer perception, and the audio feature MFCC, MEL, chroma, Tonnetz were used. These models have been trained to recognize these emotions (Calm, neutral, surprise, happy, sad, angry, fearful, disgust). We got an accuracy of 86.5% and testing it with the input audio we get the same.

Keywords

Detection, Speech Input, Feature Extraction

            To detect the emotions that are evoked by the person who speaks is the major goal of our study. Speech and emotion detection is very crucial and it also helpful in human machine communication. To detect the emotions, we are utilizing various classifiers including SVM, Multi-layer perception, and also used the audio features such as MFCC, MEL, chroma, Tonnetz. Emotions like Calm, neutral, happy, sad etc., are detected by training of models.

6. Speech Emotion Recognition Using Bagged Support Vector Machines

Abstract

            Speech emotion popularity is one of the quite promising and thrilling issues in the area of human computer interaction. It has been studied and analysed over several decades. It’s miles the technique of classifying or identifying emotions embedded inside the speech signal. Current challenges related to the speech emotion recognition when a single estimator is used are difficult to build and train using HMM and neural networks, Low detection accuracy, High computational power and time. In this work we executed emotion category on corpora — the berlin emodb, and the ryerson audio-visible database of emotional speech and track (Ravdess). A mixture of spectral capabilities was extracted from them which changed into further processed and reduced to the specified function set. When compared to single estimators, ensemble learning has been shown to provide superior overall performance. We endorse a bagged ensemble model which consists of support vector machines with a gaussian kernel as a possible set of rules for the hassle handy. Inside the paper, ensemble studying algorithms constitute a dominant and state-of-the-art approach for acquiring maximum overall performance.

Keywords

Human-computer interaction, computational Para linguistics

            Our study implemented emotion classification by utilizing various datasets that is related to emotional speech. From the dataset, a combination of spectral capabilities was retrieved and minimized to an appropriate function set after several processes. We offered a gathered ensemble framework that contains SVM with gaussian kernel. Ensemble learning framework is compared with other existing models in our study and achieved better results.

7. Real-time Speech Emotion Detection using Artificial Intelligence

Abstract

            The study of emotions has become a significant field of study that offers potential insights for a range of applications. Man-machine interfaces must be able to detect the user’s emotional state and respond appropriately. Real-time applications of automatic emotion recognition include detecting the emotions of mobile phone users, call center operations, car drivers, pilots, etc. Emotions like happiness, joy, anger, sadness, neutral, boredom, disgust, fear and surprise can be predicted with the use of conventional algorithms such as the LSTM, Bayesian network using the Maximum Likelihood Estimation Principle and Support Vector Machine is proposed. MFCC (Mel-Frequency Cepstral Coefficients), Hamming/Hanning windows, Mel-Filter bank model were used in the proposed novel LSTM model. Implementation of novel LSTM model was performed and yielded an accuracy of 98(%). A dynamic webpage that can capture an individual’s voice and predict their emotional state was implemented using the same.

Keywords

NLP, Bayesian Network

            Maximum Likelihood prediction Principle and SVM is recommended in our approach to forecast various emotions such as happiness, joy, anger, sadness, neutral, boredom, disgust, fear and surprise by the utilization of traditional methods like LSTM and Bayesian network. Various models including MFCC, Mel-Filter bank and Hamming/Hanning windows are employed in our recommended framework and a webpage is also created to forecast emotional state of individual.

8. Machine learning techniques for speech emotion recognition using paralinguistic acoustic features

Abstract

            Speech emotion recognition is one of the fastest growing areas of interest in the field of affective computing. Emotion detection aids human–computer interaction and finds application in a wide gamut of sectors, ranging from healthcare to retail to education. The present work strives to provide a speech emotion recognition framework that is both reliable and efficient enough to work in real-time environments. Speech emotion recognition can be performed using linguistic as well as paralinguistic aspects of speech; this work focuses on the latter, using non-lexical or paralinguistic attributes of speech like pitch, intensity and Mel-frequency Cepstral coefficients to train supervised machine learning models for emotion recognition. A combination of prosodic and spectral features is used for experimental analysis and classification is performed using algorithms like Gaussian Naive Bayes, Random Forest, k-Nearest Neighbors, Support Vector Machine and Multilayer Perceptron. The choice of these ML models was based on the swiftness with which they could be trained, making them more suitable for real-time applications. Comparative analysis of the models reveals SVM and MLP to be the best performers with 77.86% and 79.62% accuracies respectively. The performance of these classifiers is compared with benchmark results in literature, and a significant improvement over state-of-the-art models is presented. The observations and findings of this work can be applied to design real-time emotion recognition frameworks that can be used to design and develop applications and technologies for various domains.

Keywords

Emotion recognition, Affective computing, Multilayer perceptron, Paralinguistic acoustic features

            A most secured and efficient model is proposed in our article to forecast the person’s speech emotions. We can utilize linguistic and paralinguistic factors, to detect speech emotions. In our article, we are using paralinguistic factors to train ML model. Categorization of speech emotions are carried out by employing Gaussian Naive Bayes, Random Forest, k-Nearest Neighbors, Support Vector Machine and Multilayer Perceptron.

9. Machine learning approach of speech emotions recognition using feature fusion technique

Abstract

            In advancement of machine learning aspect, speech based emotional states identification must have a profound impact on artificial intelligence. Proper feature selection performs a vital role on such emotion recognition. Therefore, feature fusion technology has been proposed in this study for obtaining high prediction accuracy by prioritizing the extraction of sole features. Mel Frequency Cepstral Coefficients (MFCC), Linear Predictive Coefficients (LPC), energy, Zero Crossing Rate (ZCR) and pitch are extracted and four different models are constructed for experimenting the impact of feature fusion techniques on four standard machine learning classifier namely Support Vector Machine (SVM), Linear Discriminative Analysis (LDA), Decision-Tree (D-Tree) and K Nearest Neighbour (KNN). Successful application of feature fusion techniques on our proposed classifiers give satisfactory recognition rate 96.90% on the Bengali (Indian Regional language) based dataset SUST Bangla Emotional Speech Corpus (SUBESCO), 99.82% on Toronto Emotional Speech Set (TESS) (English), 95% on Ryerson Audio-Visual Database for Emotional Speech and Song (RAVDEES) (English) and 95.33% on Berlin Database of Emotional Speech (EMO-DB) (Berlin) dataset. The presented model indicates that the proper fusion of features has a positive impact on emotion detection systems by increasing their accuracy and applicability.

Keywords

Linear predictive coefficients, linear discriminative analysis

            A method is suggested in this study that is based on feature fusion is utilized to enhance the prediction accuracy by giving importance to extraction of single feature. Different features are extracted and various frameworks are developed to analyze the effect of feature fusion method on several ML approaches like SVM, LDA, D-Tree and KNN. We conclude that, the appropriate feature fusion method provides optimal emotion detection framework.

10. Speech emotion recognition by using complex MFCC and deep sequential model

Abstract

            Speech Emotion Recognition (SER) is one of the front-line research areas. For a machine, inferring SER is difficult because emotions are subjective and annotation is challenging. Nevertheless, researchers feel that SER is possible because speech is quasi-stationery and emotions are declarative finite states. This paper is about emotion classification by using Complex Mel Frequency Cepstral Coefficients (c-MFCC) as the representative trait and a deep sequential model as a classifier. The experimental setup is speaker independent and accommodates marginal variations in the underlying phonemes. Testing for this work has been carried out on RAVDESS and TESS databases. Conceptually, the proposed model is erogenous towards prosody observance. The main contributions of this work are of two-folds. Firstly, introducing conception of c-MFCC and investigating it as a robust cue of emotion and there by leading to significant improvement in accuracy performance. Secondly, establishing correlation between MFCC based accuracy and Russell’s emotional circumplex pattern. As per the Russell’s 2D emotion circumplex model, emotional signals are combinations of several psychological dimensions though perceived as discrete categories. Results of this work are outcome from a deep sequential LSTM model. Proposed c-MFCC is found to be more robust to handle signal framing, informative in terms of spectral roll off, and therefore put forward as an input to the classifier. For RAVDESS database the best accuracy achieved is 78.8% for fourteen classes, which subsequently improved to 91.6% for gender, integrated eight classes and 98.5% for affective separated six classes. Though, the RAVDESS dataset has two analogous sentences revealed results are for the complete dataset and without applying any phonetic separation of the samples. Thus, proposed method appears to be semi-commutative on phonemes. Results obtained from this study are presented and discussed in forms of confusion matrices.

Keywords

Emotion circumplex, 1-D CNN

            Our research suggested a system for emotion categorization by utilizing deep sequential LSTM model as a classification model and considered c-MFCC as a major feature. This work has two phases, in the first phase, by providing the idea of c-MFCC, it is considered as an important feature and it leads to greater and efficient performance. In the second phase, relationship among MFCC and Russell’s emotional circumplex pattern is demonstrated.