CN118859113A - Convolutional neural network sound source localization method and system based on arrival time difference - Google Patents

Abstract

The present invention relates to the field of intelligent positioning technology, and specifically to a convolutional neural network sound source positioning method and system based on arrival time difference, the method comprising obtaining a sound source data set through a sound source array receiver; extracting the time delay characteristics of the sample data set according to a weighted phase change generalized cross-correlation algorithm; using the time delay characteristics as input, the coordinate value of the sound source as output, and using the mean square error as the loss function to train the deep residual convolutional network; obtaining the trained deep residual convolutional network; inputting the time delay characteristics corresponding to the sound source to be located into the trained deep residual convolutional network to obtain the coordinate value of the sound source. The present invention fully learns the time difference characteristics of audio data by introducing a deep residual neural network, and then comprehensively utilizes the time difference relationship, position space characteristics and data change law between multiple microphone arrays, so as to achieve better sound source positioning effect and higher accuracy.

Description

Convolutional neural network sound source localization method and system based on arrival time difference

Technical Field

The present invention relates to the field of intelligent positioning technology, and in particular to a convolutional neural network sound source positioning method and system based on arrival time difference.

Background Art

Sound source localization is one of the key indicators for evaluating the performance of acoustic systems and is widely used in various fields, including drone navigation, intelligent monitoring systems, and military applications. In-depth research on sound source localization technology involves reconstructing the sound source position in limited sensor position data and predicting the dynamic changes of the sound source in space through high-precision signal processing and algorithms, which has important application value for target positioning and tracking. Sound source localization technology is one of the advanced audio signal processing technologies. Sound source localization technology can be divided into two categories, namely, sound array (also called microphone array or microphone array) sound source localization and sound intensity probe sound field test. At present, sound source localization often uses microphones as sound source receiving sensors. Therefore, sound source localization based on microphone arrays has been widely used in voice conferencing, security, and robot hearing. It is an important part of intelligent signal processing systems. Especially in the research of acoustic signal processing, information detection and positioning, wireless communication, etc., the development of sound source localization technology is of great significance to improving system performance and expanding application fields.

With the development of deep learning, the powerful representation ability of deep neural networks has been studied in the field of sound source localization. Those skilled in the art use deep neural networks to localize sound sources. First, the spatial features of the sound source in the environment are extracted through a convolutional neural network (CNN). These features are combined with the acoustic information of the environment, and then the sound source position is accurately predicted through a fully connected neural network. The prior art discloses a hybrid neural network based on a convolutional neural network and a long-short-term memory neural network (LSTM). The hybrid network is used to learn the temporal evolution and spectral characteristics of the sound signal to achieve accurate prediction of the sound source position, thereby improving the effectiveness of the method in processing complex sound sources and changing environments. However, the above method ignores the amplitude difference between the signals received by the sound array elements, and does not fully consider the time difference relationship, positional spatial characteristics and data change laws of multiple microphone arrays, resulting in unsatisfactory sound source localization effects.

Summary of the invention

In order to solve the above technical problems, the present invention mainly aims at the difficulty that the existing sound source localization method fails to fully consider the time difference relationship, position space characteristics and data change law of multiple microphones, resulting in less than ideal sound source localization effect; the present invention provides a convolutional neural network sound source localization method based on arrival time difference and its system, which can effectively predict higher-precision sound source position coordinates in limited sound source audio data.

The first object of the present invention is to provide a convolutional neural network sound source localization method based on arrival time difference, comprising:

Acquire a sound source data set through a sound source array receiver, and perform normalization processing on the sound source data set to obtain a sample data set;

Extract the time delay characteristics of the sample data set based on the weighted phase change generalized cross-correlation algorithm;

Taking the time delay feature as input, the coordinate value of the sound source as output, and using the mean square error as the loss function, the deep residual convolutional network is trained; obtaining the trained deep residual convolutional network;

The time delay features corresponding to the sound source to be located are input into the trained deep residual convolutional network to obtain the coordinate value of the sound source.

Preferably, during the training process of the deep residual convolutional network, the time delay features are input into the deep residual convolutional network. First, the time domain features are extracted through the feature extraction module, and then the convolution module is used to mine the spatial information of the sound array. The high-order semantic information of the sound source signal is obtained through the residual module, and then transmitted to the fully connected module to output the coordinate value of the sound source.

Preferably, the fully connected module is composed of a maximum pooling layer and a fully connected layer.

Preferably, the feature extraction module is composed of a framing processing layer composed of a Hamming window with a length of 512 and a short-time Fourier transform layer after framing;

The convolution module is composed of a convolution layer, a normalization layer and a maximum pooling layer;

The residual module is composed of multiple basic residual blocks.

Preferably, in the convolution module, the convolution layer size is (64,7,7), and the step size is 2 for the two-dimensional convolution; the normalization layer performs batch normalization on the input four-dimensional array without changing the input dimension; the maximum pooling layer pooling kernel is 3*3, and the step size is 2.

Preferably, the coordinate values of the sound source are three-dimensional Cartesian coordinates.

Preferably, when acquiring the sound source data set, the abnormal data is cleaned, wherein the abnormal data includes abnormal measurement point coordinates, abnormal measurement point values, and abnormal data dimensions.

The second object of the present invention is to provide a convolutional neural network sound source localization system based on arrival time difference, comprising:

A data acquisition module is used to obtain a sound source data set through a sound source array receiver, and normalize the sound source data set to obtain a sample data set;

A model acquisition module, used for extracting the time delay characteristics of the sample data set according to the weighted phase change generalized cross-correlation algorithm;

Taking the time delay feature as input, the coordinate value of the sound source as output, and using the mean square error as the loss function, the deep residual convolutional network is trained; obtaining the trained deep residual convolutional network;

The sound source localization module is used to input the time delay features corresponding to the sound source to be located into the trained deep residual convolutional network to obtain the coordinate value of the sound source.

The present invention has at least the following beneficial effects:

The present invention provides a convolutional neural network sound source localization method and system based on arrival time difference. The method introduces deep residual convolutional neural network training to learn and capture the time domain characteristics of audio signals and the spatial information of sound arrays, and maps sample data to a more effective feature space, so that the model's generalization ability and adaptability are higher than those of traditional manually designed feature extractors.

The present invention designs a simple and effective shallow feature extraction network and network training strategy, and its training speed is much faster than other deep network feature extraction algorithms.

The present invention introduces a deep residual neural network to fully learn the time difference characteristics of audio data, and then comprehensively utilizes the time difference relationship, position space characteristics and data change rules among multiple microphone arrays, so as to achieve better sound source positioning effect and higher accuracy.

The feature selection proposed in the present invention reduces the influence of noise on the model by removing feature noise, and has relatively better robustness in a noisy environment.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a convolutional neural network sound source localization method based on time difference of arrival provided by the present invention;

FIG2 is a diagram of the deep residual convolutional network training process of the present invention;

FIG3 is a diagram of the sound source localization process of the present invention;

FIG. 4 is a two-dimensional layout diagram of microphones of the present invention, wherein FIG. 4( a ) is a two-dimensional layout diagram of multiple microphone array measurement points, and FIG. 4( b ) is a layout diagram of microphone arrays within each measurement point of a single microphone array;

FIG. 5 is an estimated error diagram of the microphone array simulation results of the present invention, wherein FIG. 5( a ) is an estimated error diagram of the simulation results of a single microphone array, and FIG. 5( b ) is an estimated error diagram of the simulation results of eight microphone arrays;

FIG6 is a visualization diagram of the sound source localization prediction result on the test data set of the present invention, wherein FIG6(a) is a two-dimensional visualization diagram of the sound source localization prediction result, and FIG6(b) is a three-dimensional visualization diagram of the sound source localization prediction result;

FIG. 7 is a diagram showing the cumulative error of the number of microphone arrays of the present invention.

DETAILED DESCRIPTION

In order to illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following is a detailed description in conjunction with embodiments.

In order to overcome the difficulty that the existing sound source localization method fails to fully consider the time difference relationship, position space characteristics and data change law of multiple microphones, the sound source localization effect is not ideal. The present invention provides a deep residual convolutional neural network sound source localization method and system based on time difference of arrival (TDOA). The method first uses the weighted phase change generalized cross-correlation (Generalized Cross-Correlation with PhaseTransform, GCC-PHAT) delay algorithm to extract the time delay characteristics of the sound array audio signal; secondly, the deep residual convolutional neural network (Deep Residual Convolution Network, DRCN) is introduced to learn the position space characteristics of the audio data, and then the feature information such as the time difference space relationship and the data change law is comprehensively used to predict the location of the sound source. Finally, the method constructs a prediction loss and trains the sound source localization model by inputting the audio data of multiple microphones, so that it can quickly predict the higher precision sound source position coordinates of different scenes. The method of the present invention can effectively predict the higher precision sound source position coordinates in limited sound source audio data.

As shown in FIG1 , a convolutional neural network sound source localization method based on arrival time difference includes:

S1, obtaining a sound source data set through a sound source array receiver, and normalizing the sound source data set to obtain a sample data set;

It should be noted that the audio signal transmitted by the sound source is received by the sound source array receiver, and the coordinate position of the corresponding sound source is recorded to produce a sound source data set.

When acquiring the sound source data set, the abnormal data is cleaned, where the abnormal data includes abnormal measurement point coordinates, abnormal measurement point values, and abnormal data dimensions.

In this embodiment, the data preprocessing process is to first clean the data set for anomalies, then divide the cleaned data into a training set, a test set, and a validation set for subsequent model training, testing, and validation, and finally perform normalization processing.

Data cleaning: Clean the abnormal data points in the original data, including but not limited to abnormal measurement point coordinates, abnormal measurement point values, and abnormal data dimensions;

Data division: Divide the cleaned data into training set, test set and validation set in a ratio of 7:2:1 for subsequent model training, testing and validation;

Data normalization processing. Specifically, all the read audio parameter values are normalized to This method helps to speed up the convergence of the network model. The specific normalization is expressed as ,in, Respectively represent the start time and end time of the audio signal recording, and the actual three-dimensional Cartesian coordinates corresponding to them As a sample to create a sample data set .

S2, extracting the time delay characteristics of the sample data set according to the weighted phase change generalized cross-correlation algorithm;

The delay feature extraction process is to firstly transform the training samples The time delay features are obtained by combining the weighted phase change generalized cross-correlation algorithm (GCC-PHAT), and then the time delay features are input into the deep residual convolutional network to learn and obtain high-order semantic features such as time domain features and acoustic array spatial information.

Delay feature extraction, input training samples The delay characteristics are obtained by combining the weighted phase change generalized cross-correlation algorithm (GCC-PHAT). Specifically, GCC-PHAT is expressed as

In the formula, represents the PHAT weighted window function, yes and The cross power spectrum function is After the delay feature extraction of all samples is completed, the delay feature set is obtained. ; is the frequency; is the time delay, which represents the time difference between the two signals; this formula converts one of the audio signals As a basis, other audio signals First, a conjugate operation is performed, and then weighted convolution is performed with the basic signal in turn to obtain the cross-power spectrum between the microphone signals, and finally the required time delay characteristics are obtained through short-time inverse Fourier transform.

S3, taking the time delay feature as input, taking the coordinate value of the sound source as output, using the mean square error as the loss function, training the deep residual convolutional network; obtaining the trained deep residual convolutional network;

During the training process of the deep residual convolutional network, the time delay features are input into the deep residual convolutional network. First, the time domain features are extracted through the feature extraction module, and then the convolution module is used to mine the spatial information of the sound array. The high-order semantic information of the sound source signal is obtained through the residual module, and then transmitted to the fully connected module to output the coordinate value of the sound source.

The coordinate values of the sound source are three-dimensional Cartesian coordinates.

The fully connected module is composed of a maximum pooling layer and a fully connected layer.

The feature extraction module is composed of a frame processing layer composed of a Hamming window with a length of 512 (and a frame shift of half the window length) and a short-time Fourier transform layer after framing.

The convolution module is composed of a convolution layer, a normalization layer and a maximum pooling layer, wherein the convolution layer size is (64,7,7), a two-dimensional convolution with a step size of 2, that is, a two-dimensional convolution with 64 dimensions and a convolution kernel size of 7*7, connected to a batch normalization layer and a maximum pooling layer; the normalization layer performs batch normalization on the input four-dimensional array without changing the input dimension; finally, the maximum pooling layer has a pooling kernel of 3*3 and a step size of 2.

The residual module is composed of multiple basic residual blocks. Each basic residual block has an ordinary convolutional neural network structure, but a cross-layer connection is introduced between the input and output of each basic block so that information can be directly transferred from the input to the output.

As shown in FIG2 , the process of extracting time domain features and acoustic array spatial information is to firstly transform the time delay features of the training samples into Input into the deep residual convolutional neural network, first use the feature extraction module to extract the time domain features of the input and obtain the time domain features Secondly, the time domain features are fused with the spatial information of the acoustic array and input into the convolution module and the residual module in turn to obtain high-order semantic features. Finally, the high-order semantic features are input into the fully connected layer to obtain the sound source localization prediction output. , and optimizes and solves it by minimizing the prediction loss objective function, making the features extracted by the network more effective.

In this embodiment, the time domain features and acoustic array spatial information are extracted. The obtained time delay features are input into the deep residual convolutional neural network. First, the time domain features are extracted through the feature extraction module, and then the convolution module is used to mine the acoustic array spatial information while reducing parameters and calculations. Finally, the high-order semantic information of the sound source signal is obtained through the residual module. Specifically, the process is expressed as ,in , , They represent feature extraction module, convolution module and residual module respectively.

Deep residual convolutional network training effectively captures the temporal and spatial dependencies between microphone arrays through network training and improves the model's generalization ability.

The network output, after passing through the residual module, the relevant features are input into a fully connected module consisting of a maximum pooling layer and a fully connected layer. The result of the pooling layer is flattened into a long vector, summarizing the underlying information and features obtained by the previous convolutional layer and pooling layer, and outputting the three-dimensional Cartesian coordinate values representing the x, y, and z axes. Specifically expressed as ,in Represents a fully connected layer module.

Loss function. The mean square error (MSE) is used as the loss function of the deep residual convolutional neural network to complete the training of the network, which is specifically expressed as ,in and Represents the sound source signal The predicted and true Cartesian coordinates of .

S4. Input the time delay features corresponding to the sound source to be located into the trained deep residual convolutional network to obtain the coordinate value of the sound source.

As shown in Figure 3, the sound source localization prediction process is to first extract the arrival time difference of multiple "microphone pair" signals through the microphone array, calculate their generalized cross-correlation function and thereby obtain the time difference relationship between multiple microphone data, then use the trained network to spatially infer the features, infer the spatial position of the unknown sound source, and obtain the spatial dependency relationship between the microphone arrays, and finally use the trained network for sound source localization prediction.

In this embodiment, the positioning error index defines the absolute error of the coordinate measurement To describe the accuracy of positioning, let the actual coordinates of the sound source be , the estimated location of the sound source is , is defined as:

The present invention is a research method for a deep residual convolutional neural network sound source localization algorithm based on arrival time difference, which can comprehensively utilize characteristic information such as the time difference relationship, position space characteristics and data change law of multiple microphones to reasonably predict the sound source position. In terms of feature extraction, a deep residual convolutional neural network is introduced to learn and capture time domain features and sound array spatial information. In terms of the sound array spatial dependency, a generalized cross-correlation function and a deep residual convolutional neural network are introduced for spatial inference, so that the method can accurately predict the sound source position with limited measurement point data.

In order to illustrate the relevant effects of the method provided by the present invention, further explanation is given through the following simulation experiment in conjunction with the accompanying drawings.

1. Simulation conditions

The present invention is a simulation performed using the Python programming language on an Intel® i5-11500 2.70GHz CPU, 16G memory, and a WINDOWS11 operating system.

The data used in the experiment is based on real data. In the 200m×200m×100m experimental area, simulated data generation experiments were used to obtain audio signal data and sound source coordinate positions of 2000 training sample sets and 500 test sample sets for subsequent network training and testing.

See FIG. 4 , where FIG. 4 (a) is a two-dimensional layout diagram of eight microphone array measurement points selected in the area, and FIG. 4 (b) is a microphone array layout diagram of an L-shaped microphone array with 9 microphones arranged in each measurement point.

2. Simulation content

A position is randomly generated in the area surrounded by the measurement points, and the sound source audio data is placed at this position to simulate the real position of the sound source. In order to verify the stability of the simulation results of the sound source localization experiment, this experimental simulation randomly generates the sound source position 10 times and conducts 10 sound source localization simulation experiments.

As shown in Figure 5, Figure 5(a) is the simulation result of using only a single microphone array, and Figure 5(b) is the simulation result of measuring point sound source localization using an 8-microphone array. Using multiple microphone array signals has smaller errors than using only a single microphone array signal, and the sound source position estimation result is more stable.

In order to further verify the effectiveness of using multiple microphone array signals, the actual position of the random sound source and the sound source position estimated by the algorithm in the above multi-microphone array experiment simulated 10 times are visualized.

Referring to FIG. 6 , FIG. 6( a ) is a two-dimensional visualization of the sound source localization prediction result, and FIG. 6( b ) is a three-dimensional visualization of the sound source localization prediction result. The sound source position predicted by the present invention is substantially consistent with the actual position.

Sound source localization prediction. Calculate the absolute error of the test sample , used to compare the error between the estimated sound source position and the actual position, and the results are shown in Table 1. Among them, serial numbers 1-8 are the absolute errors between the sound source position estimated by the microphone array and the actual position. Each number of microphone arrays conducts 10 simulation experiments. In order to facilitate the quantitative display of the experimental results of each number of microphone arrays, the maximum absolute error in the ten experiments of each number of microphone arrays is taken as the experimental display result.

As shown in Table 1 and Figure 7, as the number of microphone arrays increases, the experimental cumulative error continues to decrease and tends to be stable. Combined with the experimental results in Table 1, the number of microphones can be set to 5 or 6, which can reduce the cost of microphone experiment while ensuring the accuracy of sound source localization and controlling the calculation overhead.

Table 1 Actual and estimated sound source locations and error results

In summary, the present invention meets the positioning error accuracy index and achieves better results when predicting sound source localization. The present invention combines the traditional audio signal processing algorithm with deep learning, introduces the weighted phase change generalized cross-correlation algorithm to perform preliminary time delay feature extraction on the audio signal, and then introduces the deep residual convolutional neural network to combine it to learn and capture high-order semantic information such as time domain features and sound array spatial information in the audio signal. The present invention can fully extract the key features in the sound array audio signal, and is faster and more accurate in sound source localization prediction, and the effectiveness of the present invention has been verified through the above simulation experiments.

The present invention provides a convolutional neural network sound source localization system based on arrival time difference, comprising:

A data acquisition module is used to obtain a sound source data set through a sound source array receiver, and normalize the sound source data set to obtain a sample data set;

A model acquisition module, used for extracting the time delay characteristics of the sample data set according to the weighted phase change generalized cross-correlation algorithm;

Taking the time delay feature as input, the coordinate value of the sound source as output, and using the mean square error as the loss function, the deep residual convolutional network is trained; obtaining the trained deep residual convolutional network;

The sound source localization module is used to input the time delay features corresponding to the sound source to be located into the trained deep residual convolutional network to obtain the coordinate value of the sound source.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims (8)

1. The convolutional neural network sound source localization method based on the arrival time difference is characterized by comprising the following steps of:

Acquiring a sound source data set through a sound source array receiver, and carrying out normalization processing on the sound source data set to acquire a sample data set;

extracting time delay characteristics of a sample data set according to a weighted phase change generalized cross correlation algorithm;

Taking the time delay characteristic as input, taking the coordinate value of the sound source as output, and training the depth residual convolution network by taking the mean square error as a loss function; acquiring a trained depth residual convolution network;

And inputting the time delay characteristics corresponding to the sound source to be positioned into a trained depth residual convolution network, and obtaining the coordinate value of the sound source.

2. The arrival time difference-based convolutional neural network sound source localization method according to claim 1, wherein in the training process of the depth residual convolutional network, time delay characteristics are input into the depth residual convolutional network, time domain characteristic extraction is firstly carried out through a characteristic extraction module, then acoustic array space information is mined through a convolutional module, high-order semantic information of a sound source signal is obtained through a residual module, and after the high-order semantic information is transmitted to a full-connection module, coordinate values of the sound source are output.

3. The arrival time difference based convolutional neural network sound source localization method of claim 2, wherein the fully connected module is comprised of a max pooling layer and a fully connected layer.

4. The arrival time difference-based convolutional neural network sound source localization method of claim 2, wherein the feature extraction module is composed of a framing processing layer consisting of hamming windows with length of 512 and a short-time fourier transform layer after framing;

the convolution module consists of a convolution layer, a normalization layer and a maximum pooling layer;

the residual block is composed of a plurality of basic residual blocks.

5. The arrival time difference based convolutional neural network sound source localization method of claim 4, wherein in said convolutional module, said convolutional layer is a two-dimensional convolution of size (64,7,7) with a step size of 2; the normalization layer performs batch normalization processing on the input four-dimensional array without changing the input dimension; the maximum pooling layer pooling core is 3*3, and the step length is 2.

6. The arrival time difference based convolutional neural network sound source localization method of claim 1, wherein the coordinate values of the sound source are three-dimensional cartesian coordinates.

7. The arrival time difference based convolutional neural network sound source localization method of claim 1, wherein the anomaly data is cleaned when a sound source data set is acquired, wherein the anomaly data comprises a measurement point coordinate anomaly, a measurement point numerical anomaly, and a data dimension anomaly.

8. A convolutional neural network sound source localization system based on time differences of arrival, comprising:

The data acquisition module is used for acquiring a sound source data set through the sound source array receiver, and carrying out normalization processing on the sound source data set to acquire a sample data set;

The model acquisition module is used for extracting the time delay characteristics of the sample data set according to a weighted phase change generalized cross correlation algorithm;

Taking the time delay characteristic as input, taking the coordinate value of the sound source as output, and training the depth residual convolution network by taking the mean square error as a loss function; acquiring a trained depth residual convolution network;

And the sound source positioning module is used for inputting the time delay characteristics corresponding to the sound source to be positioned into the trained depth residual convolution network to acquire the coordinate value of the sound source.