Merge pull request #34 from jsingh811/paper-update

Update paper

Author: jsingh811

README.md

@@ -185,6 +185,25 @@ python pyAudioProcessing/extract_features.py -f "data_samples/testing"  -feats "
 ```  
 Features extracted get saved in `audio_features.json`.  
 
+## Audio format conversion
+
+You can convert you audio in `.mp4`, `.mp3`, `.m4a` and `.aac` to `.wav`. This will allow you to use audio feature generation and classification functionalities.
+
+In order to convert your audios, the following code sample can be used.  
+
+```
+from pyAudioProcessing.convert_audio import convert_files_to_wav
+
+# dir_path is the path to the directory/folder on your machine containing audio files
+dir_path = "data/mp4_files"
+
+# simple change audio_format to "mp3", "m4a" or "acc" depending on the format
+# of audio that you are trying to convert to wav
+convert_files_to_wav(dir_path, audio_format="mp4")
+
+# the converted wav files will be saved in the same dir_path location.
+
+```
 
 ## Author  
 

paper/paper.bib

@@ -64,3 +64,39 @@ @article{giannakopoulos2015pyaudioanalysis
   year={2015},
   publisher={Public Library of Science}
 }
+
+@phdthesis{phdthesis,
+  author = {Dinger, Vincent},
+  year = {2021},
+  month = {03},
+  pages = {},
+  title = {Master Thesis KI Methodiken für die Verarbeitung akustischer Signale AI Usage for Processing Acoustic Signals},
+  doi = {10.13140/RG.2.2.15872.97287}
+}
+
+@inbook{packt,
+  author = {Ben Auffarth},
+  year = {2020},
+  month = {10},
+  title = {Artificial Intelligence with Python Cookbook},
+  isbn = {9781789133967},
+}
+
+@misc{tzanetakis_essl_cook_2001,
+	author    = "Tzanetakis, George and Essl, Georg and Cook, Perry",
+	title     = "Automatic Musical Genre Classification Of Audio Signals",
+	url       = "http://ismir2001.ismir.net/pdf/tzanetakis.pdf",
+	publisher = "The International Society for Music Information Retrieval",
+	year      = "2001"
+}
+
+@misc{nlp,
+	doi = {10.5281/ZENODO.4915746},
+	url = {https://zenodo.org/record/4915746},
+	author = {Singh,  Jyotika},
+	keywords = {YouTube,  NER,  NLP},
+	title = {jsingh811/pyYouTubeAnalysis: pyYouTubeAnalysis: YouTube data requests and NER on text},
+	publisher = {Zenodo},
+	year = {2021},
+	copyright = {Open Access}
+}

paper/paper.md

@@ -1,5 +1,5 @@
 ---
-title: 'pyAudioProcessing: Audio Processing, Feature Extraction and building Machine Learning Models from Audio Data'
+title: 'pyAudioProcessing: Audio Processing, Feature Extraction and Building Machine Learning Models from Audio Data'
 tags:
   - Python
   - audio
@@ -24,17 +24,87 @@ bibliography: paper.bib
 
 # Summary
 
-PyAudioProcessing is a Python based library for processing audio data, forming and extracting numerical features from audio and further building and testing machine learning models. This library allows you to extract features such as MFCC, GFCC, spectral features, chroma features and other beat based and cepstrum based features from audio to use with one's own classification backend or popular scikit-learn classifiers.
+PyAudioProcessing is a Python based library for processing audio data, constructing and extracting numerical features from audio, building and testing machine learning models and classifying data with existing pre-trained audio classification models or custom user-built models. PyAudioProcessing provides five core functionalities comprising different stages of audio signal processing.  
+
+1. Converting audio files to ".wav" format to give the users the ability to work with different types of audio files and convert them to ".wav" to increase compatibility with code and processes and work with ".wav" audio type.
+
+2. Builds numerical features from audio that can be used to train machine learning models. The set of features supported evolve with time as research informs new and improved algorithms.
+
+3. Includes the ability to export the features built with this library to use with any custom machine learning backend of the user's choosing.
+
+4. Includes the capability that allows users to train scikit-learn classifiers using features of their choosing directly from raw data. This library runs 
+
+	a. automatic hyper-parameter tuning
+	b. returns to the user the training model metrics along with cross-validation confusion matrix for model evaluation
+	c. allows the users to test the created classifier with the same features used for training
+
+5. Includes pre-trained models to provide users with baseline audio classifiers.
+
+It in an end-to-end solution for converting between audio file formats, building features from raw audio samples and training a machine learning model that can then be used to classify unseen raw audio samples. This library allows the user to extract features such as MFCC, GFCC, spectral features, chroma features and other beat based and cepstrum based features from audio to use with one's own classification backend or popular scikit-learn classifiers that have been built into pyAudioProcessing. 
+
+MATLAB is the language of choice for a vast amount of research in the audio and speech processing domain. On the contrary, Python remains the language of choice for a vast majority of Machine Learning research and functionality. This library contains features converted to Python that were originally built in MATLAB following a research invention. This software contributes to the available open-source software by enabling users to use Python based machine learning backend with highly researched audio features such as GFCC and others that are actively user for many audio classification based applications but are not readily available in Python due to primary popularity of research in MATLAB.  
+
+This software aims to provide machine learning engineers, data scientists, researchers and students with a set of baseline models to classify audio, the ability to use this library to build features on custom training data, the ability to automatically train on a scikit-learn classifier and perform hyper-parameter tuning using this library, the ability to export the built features for integration with any machine learning backend and the ability to classify audio files. This software furthers aims to aid users in addressing research efforts using GFCC and other evolving and actively researched audio features possible with Python.
+
 
 # Statement of need
 
+The motivation behind this software is understanding the popularity of Python for Machine Learning and presenting solutions for computing complex audio features using Python. This not only implies the need for resource to guide solutions for audio processing, but also signifies the need for Python guides and implementations to solve audio and speech classification tasks. The classifier implementation examples that are a part of this software and the README aim to give the users a sample solution to audio classification problems and help build the foundation to tackle new and unseen problems. 
+
+Different data processing techniques work well for different types of data. For example, word vector formations work great for text data [@nlp]. However, passing numbers data, an audio signal or an image through word vector formation is not likely to bring back any meaningful numerical representation that can be used to train machine learning models. Different data types correlate with feature formation techniques specific to their domain rather than a "one size fits all".
+
 PyAudioProcessing is a Python based library for processing audio data into features and building Machine Learning models. Audio processing and feature extraction research is popular in MATLAB. There are comparatively fewer resources for audio processing and classification in Python. This tool contains implementation of popular and different audio feature extraction that can be use in combination with most scikit-learn classifiers. Audio data can be trained, tested and classified using pyAudioProcessing. The output consists of cross validation scores and results of testing on custom audio files.
 
 The library lets the user extract aggregated data features calculated per audio file. Unique feature extractions such as Mel Frequency Cepstral Coefficients (MFCC) [@6921394], Gammatone Frequency Cepstral Coefficients (GFCC) [@inbook], spectral coefficients, chroma features and others are available to extract and use in combination with different backend classifiers. While MFCC features find use in most commonly encountered audio processing tasks such as audio type classification, speech classification, GFCC features have been found to have application in speaker identification or speaker diarization. Many such applications, comparisons and uses can be found in this IEEE paper [@6639061]. All these features are also helpful for a variety of other audio classification tasks.
 
 Some other popular libraries for the domain of audio processing include librosa [@mcfee2015librosa] and pyAudioAnalysis [@giannakopoulos2015pyaudioanalysis]. Librosa is a python package for music and audio analysis. It provides the building blocks necessary to create music information retrieval systems. PyAudioAnalysis is a python library for audio feature extraction, classification, segmentation and applications. It allows the user to train scikit-learn models for mfcc, spectral and chroma features.
 
-PyAudioProcessing adds multiple additional features. The library includes the implementation of GFCC features converted from MATLAB code to allow users to leverage features for speech classification and speaker identification tasks in addition to MFCC and spectral features that are useful for music and other audio classification tasks. It allows the user to choose from the different feature options and use single or combinations of different audio features. The features can be run through a variety of scikit-learn models including a grid search for best model and Hyperparameters, along with a final confusion matrix and cross validation performance statistics. It further allows for saving and exporting the different audio features per audio file for the user to be able to leverage those while using a different custom classifier backend that is not a part of scikit-learn's models. 
+PyAudioProcessing adds multiple additional features. The library includes the implementation of GFCC features converted from MATLAB based research to allow users to leverage Python with features for speech classification and speaker identification tasks in addition to MFCC and spectral features that are useful for music and other audio classification tasks. It allows the user to choose from the different feature options and use single or combinations of different audio features. The features can be run through a variety of scikit-learn models including a grid search for best model and Hyperparameters, along with a final confusion matrix and cross validation performance statistics. It further allows for saving and exporting the different audio features per audio file for the user to be able to leverage those while using a different custom classifier backend that is not a part of scikit-learn's models. 
+
+The library further provides some pre-build audio classification models such as `speechVSmusic`, `speechVSmusicVSbirds` sound classifier and `music genre` classifier for give the users a baseline of pre-trained models for their common audio classification tasks. The user can use the library to build custom classifiers with the help of the instructions in the README.
+
+There is an additional functionality that allows users to convert their audio files to "wav" format to gain compatibility for using analysis and feature extraction on their audio files.
+
+Given the use of this software in the community today inspires the need and growth of this software. It is referenced in a text book titled `Artificial Intelligence with Python Cookbook` published by Packt Publishing in October 2020 [@packt]. Additionally, pyAudioProcessing is a part of specific admissions requirement for a funded PhD project at University of Portsmouth <sup id="portsmouth">[1](#footnote_portsmouth)</sup>. It is further referenced in this thesis paer titled "Master Thesis AI Methodologies for Processing Acoustic Signals AI Usage for Processing Acoustic Signals" [@phdthesis].
+
+<b id="footnote_portsmouth">1</b> https://www.port.ac.uk/study/postgraduate-research/research-degrees/phd/explore-our-projects/detection-of-emotional-states-from-speech-and-text [↩](#portsmouth)
+
+
+# Pre-trained models
+
+This software offer pre-trained models. This is an evolving feature as new datasets and classification problems gain prominence in research. Some of the pre-trained models include the following.
+
+1. Audio type classifier to determine speech versus music: Trained SVM classifier for classifying audio into two possible classes - music, speech. This classifier was trained using MFCC, spectral and chroma features. Confusion matrix has scores such as follows.
+
+| | music | speech |
+| --- | --- | --- |
+| music | 48.80 | 1.20 |
+| speech | 0.60 | 49.40 |
+
+2. Audio type classifier to determine speech versus music versus bird sounds: Trained SVM classifier that classifying audio into three possible classes - music, speech and birds. This classifier was trained using MFCC, spectral and chroma features. Confusion matrix has scores such as follows.
+
+| | music | speech | birds |
+| --- | --- | --- | --- |
+| music | 31.53 | 0.73 | 1.07 |
+| speech | 1.00 |	32.33 | 0.00 |
+| birds | 0.00 | 0.00 | 33.33 |
+
+3. Music genre classifier using the GTZAN [@tzanetakis_essl_cook_2001] dataset: Trained on SVM classifier using GFCC, MFCC, spectral and chroma features to classify music into 10 genre classes - blues, classical, country, disco, hiphop, jazz, metal, pop, reggae, rock. Confusion matrix has scores such as follows.
+
+| | pop | met | dis | blu | reg | cla | rock | hip | cou | jazz |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | 
+| pop | 7.25 | 0.00 | 0.74 | 0.38 | 0.09 | 0.09 | 0.33 | 0.60 | 0.50 | 0.04 |
+| met | 0.03 | 8.74 | 0.66 | 0.09 | 0.00 | 0.00 | 0.45 | 0.00 | 0.04 | 0.00 |
+| dis | 0.69 | 0.08 | 6.29 | 0.00 | 0.74 | 0.11 | 0.90 | 0.51 | 0.69 | 0.00 |
+| blu | 0.00 | 0.20 | 0.00 | 8.31 | 0.25 | 0.08 | 0.44 | 0.09 | 0.30 | 0.34 |
+| reg | 0.11 | 0.00 | 0.26 | 0.58 | 7.99 | 0.00 | 0.28 | 0.59 | 0.09 | 0.11 |
+| cla | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 9.07 | 0.23 | 0.00 | 0.23 | 0.48 |
+| rock | 0.14 | 0.90 | 1.10 | 0.80 | 0.35 | 0.29 | 5.31 | 0.01 | 1.09 | 0.01 |
+| hip | 0.71 | 0.14 | 0.56 | 0.18 | 1.96 | 0.00 | 0.19 | 6.10 | 0.03 | 0.14 |
+| cou | 0.25 | 0.15 | 0.84 | 0.64 | 0.08 | 0.10 | 1.87 | 0.00 | 5.84 | 0.24 |
+| jazz | 0.04 | 0.01 | 0.13 | 0.41 | 0.00 | 0.76 | 0.31 | 0.00 | 0.53 | 7.81 |
+
+These baseline models aim to present capability of audio feature generation algorithms in extracting meaningful numeric patterns from the audio data. One can train their own classifiers using similar features and different machine learning backend for researching and exploring improvements.
 
 # Audio features
 

pyAudioProcessing/convert_audio.py

@@ -0,0 +1,137 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""
+Created on Thu Jun 10 15:23:55 2021
+
+@author: jsingh
+"""
+# Imports
+
+import os
+import glob
+
+from pydub import AudioSegment
+
+
+# Functions
+
+def convert_from_m4a(file_path):
+    """
+    Converts m4a audio into wav format.
+    """
+    try:
+        track = AudioSegment.from_file(file_path, "m4a")
+        file_handle = track.export(
+            file_path.replace(".m4a", ".wav"), format='wav'
+        )
+    except FileNotFoundError:
+        print("{} does not appear to be valid. Please check.")
+    except Exception as e:
+        print(e)
+
+
+def convert_from_mp3(file_path):
+    """
+    Converts mp3 audio into wav format.
+    """
+    try:
+        track = AudioSegment.from_file(file_path, "mp3")
+        file_handle = track.export(
+            file_path.replace(".mp3", ".wav"), format='wav'
+        )
+    except FileNotFoundError:
+        print("{} does not appear to be valid. Please check.")
+    except Exception as e:
+        print(e)
+
+
+def convert_from_mp4(file_path):
+    """
+    Converts mp4 audio into wav format.
+    """
+    try:
+        track = AudioSegment.from_file(file_path, "mp4")
+        file_handle = track.export(
+            file_path.replace(".mp4", ".wav"), format='wav'
+        )
+    except FileNotFoundError:
+        print("{} does not appear to be valid. Please check.")
+    except Exception as e:
+        print(e)
+
+
+def convert_from_aac(file_path):
+    """
+    Converts aac audio into wav format.
+    """
+    try:
+        track = AudioSegment.from_file(file_path, "aac")
+        file_handle = track.export(
+            file_path.replace(".aac", ".wav"), format='wav'
+        )
+    except FileNotFoundError:
+        print("{} does not appear to be valid. Please check.")
+    except Exception as e:
+        print(e)
+
+
+def convert_files_to_wav(dir_path, audio_format="m4a"):
+    """
+    Converts all the audio files in the input directory path
+    with the extension specified by audio_format input
+    into .wav audio files, and saves them in the same directory.
+    """
+    # Read contents of dir
+    # Only select files with the mentioned extension
+    files = glob.glob(os.path.join(dir_path, "*." + audio_format))
+
+    # Convert to wav
+    # The wav files save in the same dir as specified by dir_path
+    if audio_format == "m4a":
+        cntr = 0
+        for aud_file in files:
+            convert_from_m4a(aud_file)
+            cntr += 1
+        print(
+            "{} {} files converted to .wav and saved in {}".format(
+                cntr, audio_format, dir_path
+            )
+        )
+    elif audio_format == "mp3":
+        cntr = 0
+        for aud_file in files:
+            convert_from_mp3(aud_file)
+            cntr += 1
+        print(
+            "{} {} files converted to .wav and saved in {}".format(
+                cntr, audio_format, dir_path
+            )
+        )
+    elif audio_format == "aac":
+        cntr = 0
+        for aud_file in files:
+            convert_from_aac(aud_file)
+            cntr += 1
+        print(
+            "{} {} files converted to .wav and saved in {}".format(
+                cntr, audio_format, dir_path
+            )
+        )
+    elif audio_format == "mp4":
+        cntr = 0
+        for aud_file in files:
+            convert_from_mp4(aud_file)
+            cntr += 1
+        print(
+            "{} {} files converted to .wav and saved in {}".format(
+                cntr, audio_format, dir_path
+            )
+        )
+    else:
+        print(
+            "File format {} is not in supported types (mp3, mp4, m4a, aac)".format(
+                audio_format
+            )
+        )
+    
+    

pyAudioProcessing/run_classification.py

@@ -132,7 +132,7 @@ def train_and_classify(
     Train on the data under folder_path or classify the data in folder path
     using features specified by feature_names and the specified classifier.
     """
-    # Get all direcotiers under folder_path
+    # Get all directories under folder_path
     data_dirs = [x[0] for x in os.walk(folder_path)][1:]
     print(
         "\n There are {} classes in the specified data folder\n".format(