In part one, we learnt to extract various features from audio clips. We, also, trained a two layer neural network to classify each sound into a predefined category. Today, we will go one step further and see how we can apply Convolution Neural Network (CNN) to perform the same task of urban sound classification.
Note: If you want to get in-depth understanding of how CNN works. Kindly consult following resources:[1], [2] and [3].
The well-known application of CNN is image classification, where a fixed dimension image is fed into a network along with different channels (RGB in the case of a color image) and after various steps of convolution, pooling and fully connected layers, network outputs class probabilities for the image. We want to do the same, but here instead of an image, we have sound clips. A quick search on Google Scholar provide a lot of research papers, which discuss the implementation of CNN on a sound dataset. A paper I found particularly interesting and quite relevant is Environmental sound classification with convolutional neural networks by Karol J. Piczak. I borrowed the idea of dataset (feature extraction) preparation for CNN from this paper. For example: how to get equal size segments from varying length audio clips and which audio feature(s) we can feed as a separate channel (just like RGB of a color image) into the network. Once we have the initial dataset ready for the CNN. We can train as deep network (composed of different layers) as we want!
Let's define a function to calculate log scaled mel-spectrograms and their corresponding deltas from a sound clip. Regarding fixed size input, we will divide each sound clip into segments of 60x41 (60 rows and 41 columns). The mel-spec and their deltas will become two channels, which we will be fed into CNN. Other features can be calculated in the same way, which can be used as a separate input channel.
import glob
import os
import librosa
import matplotlib.pyplot as plt
import tensorflow as tf
import numpy as np
%matplotlib inline
plt.style.use('ggplot')
def windows(data, window_size):
start = 0
while start extract_feature and windows are the two methods we need to prepare the data (both features and labels) for CNN. extract_features iterates over all the files within subdirectories of a particular parent directory, calculate above-mentioned features along with class labels and append them to arrays. Let's call this method to extract features and labels and save them in corresponding variables. Also, convert labels into one hot vector using one_hot_encode method and split dataset into training and testing sets, with 70/30 ratio.
parent_dir = 'Sound-Data'
sub_dirs= ['fold1','fold2']
features,labels = extract_features(parent_dir,sub_dirs)
labels = one_hot_encode(labels)rnd_indices = np.random.rand(len(labels)) Now we define some helper functions for the implementation of CNN. The method named weight_variable and bias_variable will return Tensorflow variable of defined shapes, where bias variable is initialized with all ones and weight variable with zero mean and standard deviation of 0.1. The Conv2d method is just a wrapper over Tensorflow conv2d function. It will be called by apply_convolution function, which takes input data, kernel/filer size, a number of channels in the input and output depth or number of channels in the output. It then gets weight and bias variables, applies convolution, adds the bias to the results and finally applies non-linearity (RELU). Max-Pooling can be applied using apply_max_pool function. It takes input data (usually output of convolution layer), kernel and stride size. I used the SAME padding, you can change it to VALID padding if you want.
def weight_variable(shape):
initial = tf.truncated_normal(shape, stddev = 0.1)
return tf.Variable(initial)
def bias_variable(shape):
initial = tf.constant(1.0, shape = shape)
return tf.Variable(initial)
def conv2d(x, W):
return tf.nn.conv2d(x,W,strides=[1,2,2,1], padding='SAME')
def apply_convolution(x,kernel_size,num_channels,depth):
weights = weight_variable([kernel_size, kernel_size, num_channels, depth])
biases = bias_variable([depth])
return tf.nn.relu(tf.add(conv2d(x, weights),biases))
def apply_max_pool(x,kernel_size,stride_size):
return tf.nn.max_pool(x, ksize=[1, kernel_size, kernel_size, 1],
strides=[1, stride_size, stride_size, 1], padding='SAME')
The code provided below defines configuration parameters required by CNN model. Such as kernel size, training epochs, a number of neurons in each hidden layer and learning rate etc.
frames = 41
bands = 60
feature_size = 2460 #60x41
num_labels = 10
num_channels = 2
batch_size = 50
kernel_size = 30
depth = 20
num_hidden = 200
learning_rate = 0.01
training_epochs = 2000
Tensorflow placeholder for input and output data are defined next. A convolution function is applied with a filter size of 30 and depth of 20 (number of channels, we will get as output from convolution layer). Next, the convolution output is flattened out for the fully connected layer input. There are 200 neurons in the fully connected layer as defined by the above configuration. The Sigmoid function is used as non-linearity in this layer. Lastly, the Softmax layer is defined to output probabilities of the class labels.
X = tf.placeholder(tf.float32, shape=[None,bands,frames,num_channels])
Y = tf.placeholder(tf.float32, shape=[None,num_labels])
cov = apply_convolution(X,kernel_size,num_channels,depth)
shape = cov.get_shape().as_list()
cov_flat = tf.reshape(cov, [-1, shape[1] * shape[2] * shape[3]])
f_weights = weight_variable([shape[1] * shape[2] * depth, num_hidden])
f_biases = bias_variable([num_hidden])
f = tf.nn.sigmoid(tf.add(tf.matmul(cov_flat, f_weights),f_biases))
out_weights = weight_variable([num_hidden, num_labels])
out_biases = bias_variable([num_labels])
y_ = tf.nn.softmax(tf.matmul(f, out_weights) + out_biases)
The cross-entropy cost function will be minimised using Adam optimizer, the code provided below initialize cost function and optimizer. Also, define the code for accuracy calculation of the prediction by model.
cross_entropy = -tf.reduce_sum(Y * tf.log(y_))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y_,1), tf.argmax(Y,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))Now the following code will train the CNN model using a batch size of 50 for 2000 training epochs. After the training, it classifies testing set and prints out the achieved accuracy of the model along with plotting cost as a function of a number of iterations.
cost_history = np.empty(shape=[1],dtype=float)
with tf.Session() as session:
tf.initialize_all_variables().run()
for epoch in range(training_epochs):
offset = (epoch * batch_size) % (train_y.shape[0] - batch_size)
batch_x = train_x[offset:(offset + batch_size), :, :, :]
batch_y = train_y[offset:(offset + batch_size), :]
_, c = session.run([optimizer, cross_entropy],feed_dict={X: batch_x, Y : batch_y})
cost_history = np.append(cost_history,c)
print('Test accuracy: ',round(session.run(accuracy, feed_dict={X: test_x, Y: test_y}) , 3))
fig = plt.figure(figsize=(15,10))
plt.plot(cost_history)
plt.axis([0,training_epochs,0,np.max(cost_history)])
plt.show()
This blog post discussed how to prepare a dataset for CNN and train a model with one convolution layer. I would encourage you to train deep models with several convolution, pooling and two or more fully connected layers on complete dataset. If you have any question or feedback, please comment below.
The python notebook is available at the following link.