CN111310566A - A method and system for wildfire detection based on static and dynamic multi-feature fusion - Google Patents

Abstract

The invention discloses a method and system for detecting a mountain fire with static and dynamic multi-feature fusion. The method first performs frame sampling on the mountain fire video to generate an initial image sequence and preprocesses it to generate a preprocessed image sequence; divides the preprocessed image sequence into static samples and dynamic samples, and uses a convolutional neural network to extract all the images. Describe the static features of the static samples and extract the dynamic features of the dynamic samples; use the nuclear canonical correlation analysis method to fuse the static features and dynamic features to generate fused features; input the fused features into the support vector machine classifier for learning and training , generate a trained wildfire detection classifier; use the trained wildfire detection classifier for wildfire detection. In the process of extracting image features based on video frames, the method of the invention adopts the method of separately extracting and re-merging the static and dynamic features of the pre-processed images, which can comprehensively extract the mountain fire features and improve the accuracy and accuracy of the mountain fire detection. Spend.

Description

A method and system for wildfire detection based on static and dynamic multi-feature fusion

technical field

The present invention relates to the technical field of wildfire detection, in particular to a method and system for wildfire detection based on static and dynamic multi-feature fusion.

Background technique

With the rapid development of the power grid and the continuous expansion of the power grid structure, the power grid inevitably has to pass through the mountains and forests, so wildfires have become a situation that must be prevented when laying the power grid. Because of the complex terrain and vast area in the mountains and forests, manual monitoring is difficult, and the automatic monitoring of wildfires can greatly reduce the waste of manpower. The rapid development of image processing technology and computer vision provides a technical basis for the automatic detection of wildfires by computers.

Most of the current wildfire identification and detection methods use convolutional neural networks alone or use traditional feature extraction methods to extract features from images in moving areas. The weather conditions are also more complex, so the current detection of wildfires has shortcomings such as incomplete feature extraction and low recognition accuracy.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method and system for detecting wildfires with static and dynamic multi-feature fusion, so as to solve the problems that the existing wildfire identification and detection methods do not fully extract the characteristics of wildfires and have low identification accuracy.

For achieving the above object, the present invention provides the following scheme:

A method for detecting wildfires based on static and dynamic multi-feature fusion, the method for detecting wildfires includes:

Obtaining an initial image sequence generated by frame sampling of a mountain fire video; the initial image sequence includes a plurality of consecutively shot original images;

Preprocessing the initial image sequence to generate a preprocessed image sequence;

dividing the preprocessed image sequence into static samples and dynamic samples;

Extract the static features of the static samples by using a convolutional neural network;

extracting dynamic features of the dynamic samples;

The static feature and the dynamic feature are fused by using the kernel canonical correlation analysis method to generate the fused feature;

Inputting the fused features into a support vector machine classifier for learning and training to generate a trained wildfire detection classifier;

The mountain fire detection is performed by using the trained mountain fire detection classifier.

Optionally, the preprocessing of the initial image sequence to generate a preprocessed image sequence specifically includes:

Denoising is performed on each original image in the initial image sequence by using a median filtering method to generate a denoised image;

The single-scale Retinex method is used to perform image enhancement processing on the denoised image to generate an enhanced image; a plurality of consecutive enhanced images constitute the preprocessed image sequence.

Optionally, dividing the preprocessed image sequence into static samples and dynamic samples specifically includes:

Divide multiple continuous enhanced images in the preprocessed image sequence into training samples and test samples according to a ratio of 4:1;

The multiple consecutive enhanced images in the training samples are divided into static samples and dynamic samples in a ratio of 3:2.

Optionally, the use of a convolutional neural network to extract the static features of the static samples specifically includes:

The static sample is input into the VGG-19 convolutional neural network, and the feature map output by the VGG-19 convolutional neural network is obtained;

Perform an inner product operation between the features of different channels according to the height, width and number of channels of the feature map to obtain a feature matrix;

The feature matrix is vectorized into static features of the static samples.

Optionally, the extracting the dynamic features of the dynamic samples specifically includes:

Extract the motion area in the dynamic sample by using the single Gaussian background model method;

Extracting flame texture features using a grayscale co-occurrence matrix for the motion region;

extracting the area change feature and flicker feature of the motion area;

The flame texture feature, the area change feature, and the flicker feature are weighted and combined into a dynamic feature of the dynamic sample.

A static and dynamic multi-feature fusion wildfire detection system, the wildfire detection system includes:

an initial image sequence acquisition module, used to acquire an initial image sequence generated by frame sampling of a mountain fire video; the initial image sequence includes a plurality of consecutively shot original images;

an image preprocessing module for preprocessing the initial image sequence to generate a preprocessed image sequence;

a static and dynamic sample division module, used for dividing the preprocessed image sequence into static samples and dynamic samples;

A static feature extraction module, used for extracting the static features of the static sample by using a convolutional neural network;

a dynamic feature extraction module for extracting the dynamic features of the dynamic samples;

a static and dynamic feature fusion module, configured to fuse the static feature and the dynamic feature by using the kernel canonical correlation analysis method to generate a post-fusion feature;

a model training module for inputting the fused features into a support vector machine classifier for learning and training to generate a trained mountain fire detection classifier;

A wildfire detection module is used to detect wildfires by using the trained wildfire detection classifier.

Optionally, the image preprocessing module specifically includes:

a denoising processing unit, configured to perform denoising processing on each original image in the initial image sequence by using a median filtering method to generate a denoised image;

The enhancement processing unit is configured to perform image enhancement processing on the denoised image by using the single-scale Retinex method to generate an enhanced image; a plurality of consecutive enhanced images constitute the preprocessed image sequence.

Optionally, the static and dynamic sample division module specifically includes:

A training sample dividing unit, which is used to divide multiple continuous enhanced images in the preprocessed image sequence into training samples and test samples according to a ratio of 4:1;

The static and dynamic sample dividing unit is configured to divide the plurality of continuous enhanced images in the training samples into static samples and dynamic samples in a ratio of 3:2.

Optionally, the static feature extraction module specifically includes:

A feature map acquisition unit, used to input the static sample into the VGG-19 convolutional neural network to obtain the feature map output by the VGG-19 convolutional neural network;

a feature matrix calculation unit, configured to perform an inner product operation between features of different channels according to the height, width and number of channels of the feature map to obtain a feature matrix;

A static feature extraction unit, configured to vectorize the feature matrix into static features of the static sample.

Optionally, the dynamic feature extraction module specifically includes:

a motion region extraction unit, used for extracting the motion region in the dynamic sample by using a single Gaussian background model method;

a flame texture feature extraction unit, used for extracting flame texture features from the motion region by using a grayscale co-occurrence matrix;

an area change feature and flicker feature extraction unit for extracting the area change feature and flicker feature of the motion area;

A dynamic feature extraction unit, configured to weight the flame texture feature, the area change feature and the flicker feature into a dynamic feature of the dynamic sample.

According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

The present invention provides a method and system for detecting a mountain fire with static and dynamic multi-feature fusion. The method first performs frame sampling on a mountain fire video to generate an initial image sequence, and preprocesses the initial image sequence to generate a preprocessed image. sequence; dividing the preprocessed image sequence into static samples and dynamic samples; extracting the static features of the static samples by using a convolutional neural network and extracting the dynamic features of the dynamic samples; using the nuclear canonical correlation analysis method to The static features and the dynamic features are fused to generate the fused features; the fused features are input into the support vector machine classifier for learning and training to generate a trained wildfire detection classifier; the wildfire detection classification is adopted for wildfire detection. In the process of extracting image features based on video frames, the method of the invention adopts the method of separately extracting and re-merging the static and dynamic features of the pre-processed images, which can comprehensively extract the mountain fire features and improve the accuracy and accuracy of the mountain fire detection. Spend.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings required in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some of the present invention. In the embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative labor.

Fig. 1 is the flow chart of the wildfire detection method of static and dynamic multi-feature fusion provided by the present invention;

FIG. 2 is a schematic diagram of a method for detecting wildfires by static and dynamic multi-feature fusion provided by the present invention;

3 is a block diagram of an image preprocessing process provided by the present invention;

Fig. 4 is a flow chart of static feature extraction provided by the present invention;

Fig. 5 is a flow chart of dynamic feature extraction provided by the present invention;

FIG. 6 is a flow chart of the static and dynamic feature fusion provided by the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The purpose of the present invention is to provide a method and system for detecting wildfires with static and dynamic multi-feature fusion, so as to solve the problems that the existing wildfire identification and detection methods do not fully extract the characteristics of wildfires and have low identification accuracy.

In order to make the above objects, features and advantages of the present invention more clearly understood, the present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

FIG. 1 is a flowchart of a method for detecting wildfires by static and dynamic multi-feature fusion provided by the present invention; FIG. 2 is a schematic diagram of a method for detecting wildfires by static and dynamic multi-feature fusion provided by the present invention. Referring to FIG. 1 and FIG. 2 , the method for detecting wildfires by static and dynamic multi-feature fusion provided by the present invention specifically includes:

Step 101: Acquire an initial image sequence generated by frame sampling of the wildfire video.

The present invention performs frame sampling on the collected mountain fire video to obtain an initial image sequence; the initial image sequence includes a plurality of consecutively shot original images. The initial image sequence is divided into training samples and test samples according to a ratio of 4:1 to prepare for training the classifier later. It should be noted that when dividing training samples and test samples, 4/5 of the continuous original images are used as training samples, and the remaining 1/5 of the continuous original images are used as test samples, so as to facilitate dynamic Feature extraction.

Step 102: Preprocess the initial image sequence to generate a preprocessed image sequence.

FIG. 3 is a block diagram of an image preprocessing flow provided by the present invention. Referring to FIG. 3 , the step 102 specifically includes:

Step 2.1: Denoising each original image in the initial image sequence by using the median filter method to generate a denoised image.

All original images in the initial image sequence are preprocessed, and median filtering is used to denoise them, that is, the pixel of each point of the original image is set as the middle of all pixels in the 3×3 neighborhood with it as the center. value to generate the denoised image.

Step 2.2: using the single-scale Retinex method to perform image enhancement processing on the denoised image to generate an enhanced image; a plurality of consecutive enhanced images constitute the preprocessed image sequence.

The single-scale Retinex method is used to perform image enhancement processing on the denoised image, and the specific steps are as follows:

①Read the denoised image of each frame as the image to be enhanced I(x,y), and convert the pixel value of the image I(x,y) from an integer to a double type (double-precision floating-point type), followed by a pair of The double-type image to be enhanced I(x,y) is processed. The present invention transforms the image to be enhanced I(x, y) into a double type to facilitate the processing of subsequent data calculation.

其中r(x,y)为输出图像。②The image to be enhanced I(x,y) can be composed of the product of the reflection image R(x,y) and the incident image L(x,y): I(x,y)=R(x,y)×L(x ,y). Converting it to the logarithmic domain gives:

where r(x,y) is the output image.

(K为归一化常数)，在满足∫∫G(x,y,c)dxdy＝1的情况下取K值。③The incident image L(x,y) can be represented by the convolution of the original image I(x,y) and the Gaussian function G(x,y,c): L(x,y)=I(x,y)*G(x , y, c); where c is a Gaussian kernel with a value of 85. and the Gaussian function

(K is a normalization constant), and the value of K is taken when ∫∫G(x, y, c)dxdy=1 is satisfied.

④ After finding K, you can find r(x, y), and then convert it to the real number field.

和∫∫G(x,y,c)dxdy＝1计算K值，然后根据K值求得G(x，y，c)，进一步根据L(x,y)＝I(x,y)*G(x,y,c)求得入射图像L(x,y)，然后根据

得到输出图像r(x,y)。由于是采用对数变化

将R(x,y)转化为r(x,y)的，所以只需要反对数变换就可以将对数型态的r(x,y)转换为原来的实数域也就是R(x,y)。转化为R(x,y)也就是做了增强改变后的反射图像的像素值，也就替换了原来的像素I(x,y)，即后续I(x,y)的内容已经变成了R(x,y)，R(x,y)即为增强后图像。That is, the process of performing image enhancement processing on the denoised image in the present invention is based on the

and ∫∫G(x,y,c)dxdy=1 to calculate the K value, then obtain G(x,y,c) according to the K value, and further according to L(x,y)=I(x,y)*G (x, y, c) to obtain the incident image L(x, y), and then according to

Get the output image r(x,y). Since the logarithmic change is used

Convert R(x, y) to r(x, y), so you only need inverse logarithmic transformation to convert r(x, y) in logarithmic form to the original real number domain, which is R(x, y ). Converting to R(x,y) is the pixel value of the reflected image after the enhancement and change, which also replaces the original pixel I(x,y), that is, the content of the subsequent I(x,y) has become R(x,y), R(x,y) is the image after enhancement.

The core of Retinex is to obtain "reflection images" that represent essential information. By separating the incident image, it is possible to reduce the influence of the illumination factor on the image, enhance the detail information of the image, and obtain the content representing the essential information of the image.

The invention adopts the function of the single-scale Retinex method: Retinex can achieve a balance in three aspects of color constancy, dynamic range compression and edge enhancement, weaken the influence of uneven illumination, and enhance the detailed information of the image; and has a certain dehazing effect, There is a certain degree of tolerance for the weather conditions monitored by the field video.

So far, the image preprocessing part ends, and a plurality of consecutive enhanced images R(x, y) constitute the preprocessed image sequence.

Step 103: Divide the preprocessed image sequence into static samples and dynamic samples.

In the present invention, multiple continuous enhanced images R(x, y) in the preprocessed image sequence are divided into training samples and test samples according to the ratio of 4:1; The continuous enhanced image R(x, y) is divided into static samples and dynamic samples according to the ratio of 3:2 (it is also required to maintain video coherence) for static and dynamic feature extraction respectively.

Similarly, the test samples are also divided into static samples and dynamic samples according to the ratio of 3:2, and it is also required to maintain video coherence, so as to extract the static and dynamic features of the test samples respectively. In the subsequent training of the SVM classifier, the fused features of the training samples are used, and in the testing process, the fused features extracted from the test samples are used. Therefore, it is necessary to extract static and dynamic features for training samples and test samples separately and then fuse them, but the use of features after subsequent fusion is different.

Step 104: Extract the static features of the static samples by using a convolutional neural network.

4 is a block diagram of a static feature extraction process provided by the present invention. Referring to FIG. 4 , since the convolutional neural network is better at mining the static features of the image, but has limited ability to extract dynamic features, the present invention uses the convolutional neural network to extract all the features. Describe the static characteristics of static samples, including:

Step 4.1: Input the static sample into the VGG-19 convolutional neural network, and obtain the feature map output by the VGG-19 convolutional neural network.

The static samples in the training samples are directly input into the widely accepted VGG-19 network for processing. VGG-19 includes 16 convolutional layers, the number of channels of the convolutional layer is 64, 128, 256, 512, and finally there are three fully connected layers, the number of channels are 4096, 4096, 1000 respectively.

The features of the output of the fourth convolutional layer (the convolutional features of the lower layers) and the output features of the last fully connected layer (the convolutional features of the upper layers) are extracted respectively. The output of the convolutional neural network is denoted as a ^l , which is output in the form of a feature map [n _H , n _W , n _C ], where n _H , n _W and n _C are the height, width and number of channels of the feature map, respectively.

The effect is that the low-level convolutional features can better preserve the position and spatial information of the target itself, while the deep-level convolutional features contain more semantic information, so the extraction of the two together can make the extracted features have their own position and Spatial information and deep-level semantic information can achieve the purpose of more comprehensive feature extraction, so that the detection accuracy is higher.

Step 4.2: Perform an inner product operation between the features of different channels according to the height, width and number of channels of the feature map to obtain a feature matrix.

(k'为与k不同的另一通道)，即不同通道的特征之间进行内积运算，从而得到特征矩阵G。Let m=n _H , n=n _W , k=n _C , the texture information and color information of the picture can be extracted by using the Gram matrix, and its expression is:

(k' is another channel different from k), that is, inner product operation is performed between the features of different channels, thereby obtaining the feature matrix G.

Step 4.3: Vectorize the feature matrix into static features of the static sample.

The feature matrix G is vectorized into the static features x _st of the static samples.

This method only performs feature extraction at the same position of different channel features, so it cannot obtain enough spatial information and can only extract the global static features of the picture. The effect of separate extraction is that the low-level convolutional features can better preserve the location and spatial information of the target itself, while the deep-level convolutional features contain more semantic information.

Step 105: Extract the dynamic features of the dynamic samples.

Fig. 5 is a flow chart of dynamic feature extraction provided by the present invention. Referring to Fig. 5, the artificial dynamic feature extraction process of the present invention first uses the method of mixed Gaussian model to lock the motion area, and then extracts the specific flame texture, area change and flickering features from the motion area. . The step 105 extracts the dynamic features of the dynamic samples, specifically including:

Step 5.1: Use the single Gaussian background model method to extract the motion area in the dynamic sample; the specific method is as follows:

其中P(I(x,y,t))就是针对于图片I(x,y,t)建立的单高斯模型，μ_t和σ_t分别为t时刻该像素点(x,y)的高斯分布的期望值和标准值，μ_t为μ_t(x,y)的简写，σ_t为σ_t(x,y)的简写。a. Assume that the probability of the occurrence of the pixel value of each point of the dynamic sample image (arranged in the video order) obeys the Gaussian distribution. For the pixels in the image, I(x, y, t) is used to indicate that the pixel (x, y) is at t The pixel value at the moment, there are:

where P(I(x,y,t)) is the single Gaussian model established for the picture I(x,y,t), μ _t and σ _t are the Gaussian distribution of the pixel (x, y) at time t, respectively The expected value and standard value of , μ _t is short for μ _t (x, y), and σ _t is short for σ _t (x, y).

此时t＝0，且初始标准值σ₀(x,y)设为20。b. The background model initialized with the first frame image as the data, there are:

At this time, t=0, and the initial standard value σ ₀ (x, y) is set to 20.

c. Detect foreground and background pixels. The pixel point (x, y) that satisfies the formula |I(x,y,t)-μ _t-1 (x,y)|<T _h δ _t-1 is the foreground pixel, and it satisfies |I(x,y, t)-μ _t-1 (x, y)|≥T _h δ _t-1 The pixel point (x, y) of t-1 is the background pixel, and _Th is the set threshold (set to 0.75).

The purpose of detecting the foreground and background pixels in step c is to distinguish the foreground and the background. It can be understood that the foreground area is the moving area to be extracted by the present invention, and the background area is the static area.

其中α称为更新系数，反映了模型的更新速度：如果点(x,y)被检测为前景点，则背景模型中原来的概率分布应该得到保留，应取比较小(一般为α＝0)；如果点(x,y)被检测为背景点，则应取比较大(一般为α＝0.8)，以使背景模型中原来的概率分布能够跟上实际的变化。即对于前景点(x,y)进行背景值更新时，α取0；对于背景点(x,y)进行背景值更新时，α取0.8。d. Update the background values of μ _t , σ _t , σ _t ² , the update formula is:

Among them, α is called the update coefficient, which reflects the update speed of the model: if the point (x, y) is detected as a foreground point, the original probability distribution in the background model should be preserved and should be relatively small (usually α=0) ; If the point (x, y) is detected as a background point, it should be relatively large (generally α=0.8), so that the original probability distribution in the background model can keep up with the actual change. That is, when the background value is updated for the foreground point (x, y), α takes 0; when the background value is updated for the background point (x, y), α takes 0.8.

e. Return to step c and repeat the operation (that is, iteratively update the values of μ _t , σ _t , σ _t ² , and then send it to c to continue the comparison according to the formula, that is, to constantly compare whether it belongs to the foreground or the background), so as to update the foreground and the background model, until all the dynamic sample image sequences are run, and finally output the foreground image, which is the motion area of the present invention.

Step 5.2: Extract the flame texture feature by using the grayscale co-occurrence matrix for the motion area, which specifically includes:

a. Change the dynamic sample picture into a grayscale picture, and compress the grayscale level to 8 (generally, there are 256 grayscale levels, and the pixel value is divided by 32 into 8 grayscale levels).

b. Select a 5×5 sliding window to calculate the grayscale matrix in each direction (0°, 45°, 90°, 135°) of the compressed grayscale image (that is, each time on the image grayscale value matrix with 5×5 Window sliding observation), the number of times the gray value combination (a, b) of the two pixel points in the statistical window appears in the window, as the value of the corresponding gray co-occurrence statistical matrix at point (a, b), corresponding to 8 Each gray level produces an 8×8 gray co-occurrence matrix. The step size of each window movement is set to 1. The motion area of each frame of grayscale image gets a grayscale co-occurrence matrix.

c. Calculate the eigenvalues of the gray level co-occurrence matrix. Use P to represent the normalized frequency matrix of the grayscale co-occurrence matrix, where i, j represent the grayscale values of two levels that appear in two pixels at the same time in a certain direction, so P _ij represents two pixels that satisfy this situation probability of occurrence.

(N为灰度等级)。d. Calculate the contrast according to the probability P _ij :

(N is the gray scale).

e. Calculate the energy according to the probability P _ij :

f. The texture feature vector of the flame is: f=[f ₁ f ₂ ].

Step 5.3: Extract the area change feature and flicker feature of the motion area; the specific process is:

则火焰的面积变化特征向量为：[ΔS]。a. Suppose the area of the currently extracted motion area is S _t , the area of the motion area of the previous frame is S _t-1 , and the area is counted by the number of pixels, then the area change rate is:

Then the characteristic vector of the area change of the flame is: [ΔS].

则火焰的闪烁特征向量为：[ΔL]。b. Set the average luminance value of the currently extracted motion area to L _t , and the average luminance value of the previous frame to be L _t-1 , then the flicker feature can be represented by the change in luminance:

Then the flickering feature vector of the flame is: [ΔL].

Step 5.4: Weighting and combining the flame texture feature, the area change feature and the flicker feature into a dynamic feature of the dynamic sample.

The three groups of manually extracted dynamic features are directly weighted and summed, and the weighting coefficient is set to 1 to form a new dynamic feature y _dy , where y _dy =[f ΔS ΔL].

Step 106: Fusion of the static feature and the dynamic feature by using the kernel canonical correlation analysis method to generate a fused feature.

6 is a block diagram of a static and dynamic feature fusion process provided by the present invention. Referring to FIG. 6, the present invention utilizes a Kernel Canonical Correlation Analysis (KCCA) method to perform static and dynamic feature fusion to generate features after fusion, specifically including:

Step 6.1: Take the feature x _st extracted from VGG-19 as a static feature, directly weight and sum the three groups of manually extracted dynamic features, and set the weighting coefficient to 1 to form a new dynamic feature y _dy , where y _dy = [ f ΔS ΔL].

设核函数为

定义对应的核矩阵为

分别为核矩阵

对应的核函数。Step 6.2: Act on x _st and y _dy via two nonlinear mappings A and B:

Let the kernel function be

The corresponding kernel matrix is defined as

kernel matrix

the corresponding kernel function.

其中cov(a,b)是a和b的协方差，而D(a),D(b)分别是a和b的方差。Step 6.3: Set the vector a in the space of Α(x _st ), according to the nuclear regeneration theory, there must be a vector ξ, such that a = ξ ^T Α(x _st ); similarly, there is a vector η, such that b = η ^T Β( y _dy ). Therefore, it is only necessary to maximize the correlation coefficient ρ between a and b,

where cov(a,b) is the covariance of a and b, and D(a), D(b) are the variances of a and b, respectively.

The correlation coefficient ρ between a and b reaches the maximum value, which means that the two features to be fused have the maximum correlation in the mapped high-dimensional space, that is, the best fusion.

其中S_XY＝cov(Α(x_st),Β(y_dy))。Step 6.4: Standardize the original data (even if the mean of Α(x _st ) and β(y _dy ) is 0 and the variance is 1), then finding the maximum value of ρ can be converted into finding the maximum ξ ^T S _XY ξ and η ^T S _XY η problem, i.e.

where S _XY = cov(Α(x _st ), B(y _dy )).

Step 6.5: Introduce the Lagrangian function J(ξ, η), and the maximization problem is transformed into the maximization formula:

分别对ξ、η求导令结果为0，有：

(λ与θ为拉格朗日系数)。

Taking the derivation of ξ and η respectively, the result is 0, there are:

(λ and θ are Lagrangian coefficients).

Step 6.6: Substitute Equation (a) into Equation (b), and arrange to obtain λ=θ=ξ ^T S _XY η(c), that is, the Lagrangian coefficient is the parameter to be optimized.

其中S_XX＝cov(Α(x_st),Α(x_st))，S_YY＝cov(Β(y_dy),Β(y_dy))，S_YX＝cov(Β(y_dy),Α(x_st))。求解出ξ、η，则根据

得到a、b，即融合后的特征。Step 6.7: Substitute formula (c) into formula (b), and get

where S _XX =cov(Α(x _st ),Α(x _st )), S _YY =cov(Β(y _dy ),Β(y _dy )), S _YX =cov(Β(y _dy ),Α( x _st )). Solve for ξ, η, then according to

A and b are obtained, that is, the fused features.

Step 107: Input the fused features into a support vector machine classifier for learning and training to generate a trained mountain fire detection classifier.

First, mark the training samples as positive and negative samples, in which the positive samples are images with fire, and the negative samples are images without fire. Then perform the static and dynamic feature extraction and fusion of step 104, step 105 and step 106 on the training sample, and finally input the fused features a and b into the support vector machine (SVM) classifier for learning and training, and obtain the ability to detect wildfires. Functional SVM classifier.

The test samples are input into the SVM classifier after the same feature processing (including static and dynamic feature extraction and fusion) to detect whether there is a wildfire. The input of the SVM classifier is the fused feature, and the output is the category of the picture (with fire, no fire), and the accuracy of the classification can be known by comparing the output result with the original manual labeling result.

If the detection accuracy of the test sample is higher than the accuracy threshold, the SVM classifier will be used as the trained wildfire detection classifier; if the test sample detection accuracy is not higher than the accuracy threshold, return to step 101 to obtain the sample again. Classifier training.

Step 108: Use the trained wildfire detection classifier to detect wildfires.

The input of the wildfire detection classifier is the fused feature, and the output is the category of the picture (with fire, without fire). When performing wildfire detection, first obtain the wildfire video to be detected, perform frame sampling on the wildfire video to generate an initial image sequence; perform preprocessing on the initial image sequence to generate a preprocessed image sequence; The resulting image sequence is divided into static samples and dynamic samples; use convolutional neural network to extract the static features of the static samples and extract the dynamic features of the dynamic samples; use the nuclear canonical correlation analysis method to separate the static features and the dynamic features. The features are fused to generate the fused features of the wildfire video to be detected; the fused features are input into the trained wildfire detection classifier, and the wildfire detection result (ie, fire or no fire) can be output.

The present invention provides a method for detecting wildfires with static and dynamic multi-feature fusion. In the process of extracting image features based on video frames, a method of extracting and re-merging static and dynamic features is adopted, and finally combined with support vector machines to detect wildfires, It can extract features more comprehensively and improve the precision and accuracy of wildfire detection.

Based on the static and dynamic multi-feature fusion mountain fire detection method provided by the present invention, the present invention also provides a static and dynamic multi-feature fusion mountain fire detection system, the mountain fire detection system includes:

an initial image sequence acquisition module, used to acquire an initial image sequence generated by frame sampling of a mountain fire video; the initial image sequence includes a plurality of consecutively shot original images;

an image preprocessing module for preprocessing the initial image sequence to generate a preprocessed image sequence;

a static and dynamic sample division module, used for dividing the preprocessed image sequence into static samples and dynamic samples;

A static feature extraction module, used for extracting the static features of the static sample by using a convolutional neural network;

a dynamic feature extraction module for extracting the dynamic features of the dynamic samples;

a static and dynamic feature fusion module, configured to fuse the static feature and the dynamic feature by using the kernel canonical correlation analysis method to generate a post-fusion feature;

a model training module for inputting the fused features into a support vector machine classifier for learning and training to generate a trained mountain fire detection classifier;

A wildfire detection module is used to perform wildfire detection by using the wildfire detection classifier.

Wherein, the image preprocessing module specifically includes:

a denoising processing unit, configured to perform denoising processing on each original image in the initial image sequence by using a median filtering method to generate a denoised image;

The enhancement processing unit is configured to perform image enhancement processing on the denoised image by using the single-scale Retinex method to generate an enhanced image; a plurality of consecutive enhanced images constitute the preprocessed image sequence.

The static and dynamic sample division module specifically includes:

A training sample dividing unit, which is used to divide multiple continuous enhanced images in the preprocessed image sequence into training samples and test samples according to a ratio of 4:1;

The static and dynamic sample dividing unit is configured to divide the plurality of continuous enhanced images in the training samples into static samples and dynamic samples in a ratio of 3:2.

The static feature extraction module specifically includes:

A feature map acquisition unit, used to input the static sample into the VGG-19 convolutional neural network to obtain the feature map output by the VGG-19 convolutional neural network;

a feature matrix calculation unit, configured to perform an inner product operation between features of different channels according to the height, width and number of channels of the feature map to obtain a feature matrix;

A static feature extraction unit, configured to vectorize the feature matrix into static features of the static sample.

The dynamic feature extraction module specifically includes:

a motion region extraction unit, used for extracting the motion region in the dynamic sample by using a single Gaussian background model method;

a flame texture feature extraction unit, used for extracting flame texture features from the motion region by using a grayscale co-occurrence matrix;

an area change feature and flicker feature extraction unit for extracting the area change feature and flicker feature of the motion area;

A dynamic feature extraction unit, configured to weight the flame texture feature, the area change feature and the flicker feature into a dynamic feature of the dynamic sample.

Compared with the prior art, the static and dynamic multi-feature fusion mountain fire detection method provided by the present invention has the following advantages:

1. In step 2.2 of step 102 of the method, the single-scale Retinex method is used to perform image enhancement processing on the denoised image, because the wildfire detection is based on video shooting in the wild, and the background is more complicated. The single-scale Retinex method can enhance the detailed information of the image. , and reduce the influence of uneven illumination; there is also a certain defogging effect, which has a certain tolerance for the weather conditions of field video monitoring.

2. In steps 104 and 105 of the method, the convolutional neural network is used to extract the static features and the dynamic features are manually extracted, so that the image features can be extracted more comprehensively. Convolutional neural network is better at extracting static features of images, but has limited ability to extract dynamic features; and adding artificial dynamic extraction of motion area features can make up for this defect.

3. In step 106 of the method, Kernel Canonical Correlation Analysis (KCCA) is used for feature fusion, which can map two unrelated features to a high-dimensional space, and fuse them into a The new feature vector ensures the rationality of feature fusion.

4. The image preprocessing method (embodied in step 102) of the method combines the characteristics of the place where the fire occurs (complex background, foggy mountains and forests, etc.) to process the image, which is more practical and is also used for training the classifier. Lay a good foundation.

5. Because of the comprehensive feature extraction of the method, the accuracy of wildfire detection is greatly improved.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the description of the method.

In this paper, specific examples are used to illustrate the principles and implementations of the present invention. The descriptions of the above embodiments are only used to help understand the methods and core ideas of the present invention; meanwhile, for those skilled in the art, according to the present invention There will be changes in the specific implementation and application scope. In conclusion, the contents of this specification should not be construed as limiting the present invention.

Claims (10)

1. A mountain fire detection method based on static and dynamic multi-feature fusion is characterized by comprising the following steps:

acquiring an initial image sequence generated by performing frame sampling on a forest fire video; the initial image sequence comprises a plurality of continuously shot original images;

preprocessing the initial image sequence to generate a preprocessed image sequence;

dividing the preprocessed image sequence into static samples and dynamic samples;

extracting static characteristics of the static sample by using a convolutional neural network;

extracting dynamic features of the dynamic sample;

fusing the static features and the dynamic features by using a kernel canonical correlation analysis method to generate fused features;

inputting the fused features into a support vector machine classifier for learning training to generate a trained forest fire detection classifier;

and performing mountain fire detection by adopting the trained mountain fire detection classifier.

2. The mountain fire detection method according to claim 1, wherein the preprocessing the initial image sequence to generate a preprocessed image sequence specifically comprises:

denoising each original image in the initial image sequence by adopting a median filtering method to generate a denoised image;

performing image enhancement processing on the denoised image by adopting a single-scale Retinex method to generate an enhanced image; a plurality of successive enhanced images constitutes the pre-processed image sequence.

3. The mountain fire detection method according to claim 2, wherein the dividing the preprocessed image sequence into static samples and dynamic samples specifically comprises:

dividing a plurality of continuous enhanced images in the preprocessed image sequence into a training sample and a test sample according to a ratio of 4: 1;

dividing a plurality of continuous enhanced images in the training sample into a static sample and a dynamic sample according to a ratio of 3: 2.

4. The mountain fire detection method according to claim 3, wherein the extracting the static feature of the static sample using a convolutional neural network specifically comprises:

inputting the static sample into a VGG-19 convolutional neural network to obtain a feature map output by the VGG-19 convolutional neural network;

performing inner product operation on the features of different channels according to the height, the width and the channel number of the feature map to obtain a feature matrix;

vectorizing the feature matrix into static features of the static sample.

5. The mountain fire detection method according to claim 4, wherein the extracting of the dynamic feature of the dynamic sample specifically includes:

extracting a motion area in the dynamic sample by using a single Gaussian background model method;

extracting flame texture characteristics from the motion area by utilizing a gray level co-occurrence matrix;

extracting area change characteristics and flicker characteristics of the motion area;

and weighting and combining the flame texture feature, the area change feature and the flicker feature into the dynamic feature of the dynamic sample.

6. A mountain fire detection system with static and dynamic multi-feature fusion is characterized by comprising:

the system comprises an initial image sequence acquisition module, a frame sampling module and a frame sampling module, wherein the initial image sequence acquisition module is used for acquiring an initial image sequence generated by performing frame sampling on a forest fire video; the initial image sequence comprises a plurality of continuously shot original images;

the image preprocessing module is used for preprocessing the initial image sequence to generate a preprocessed image sequence;

a static and dynamic sample dividing module, configured to divide the preprocessed image sequence into a static sample and a dynamic sample;

the static characteristic extraction module is used for extracting the static characteristics of the static sample by utilizing a convolutional neural network;

the dynamic characteristic extraction module is used for extracting dynamic characteristics of the dynamic sample;

the static and dynamic feature fusion module is used for fusing the static features and the dynamic features by utilizing a kernel canonical correlation analysis method to generate fused features;

the model training module is used for inputting the fused features into a support vector machine classifier for learning training to generate a trained mountain fire detection classifier;

and the mountain fire detection module is used for detecting mountain fire by adopting the trained mountain fire detection classifier.

7. The wildfire detection system as claimed in claim 6, wherein the image pre-processing module specifically comprises:

the denoising processing unit is used for denoising each original image in the initial image sequence by adopting a median filtering method to generate a denoised image;

the enhancement processing unit is used for carrying out image enhancement processing on the denoised image by adopting a single-scale Retinex method to generate an enhanced image; a plurality of successive enhanced images constitutes the pre-processed image sequence.

8. The wildfire detection system as claimed in claim 7, wherein the static and dynamic sample division module specifically comprises:

a training sample dividing unit, configured to divide a plurality of continuous enhanced images in the preprocessed image sequence into a training sample and a test sample according to a ratio of 4: 1;

and the static and dynamic sample dividing unit is used for dividing the plurality of continuous enhanced images in the training sample into a static sample and a dynamic sample according to the proportion of 3: 2.

9. The wildfire detection system as claimed in claim 8, wherein the static feature extraction module specifically comprises:

the characteristic diagram acquisition unit is used for inputting the static sample into a VGG-19 convolutional neural network and acquiring a characteristic diagram output by the VGG-19 convolutional neural network;

the characteristic matrix calculation unit is used for carrying out inner product operation on the characteristics of different channels according to the height, the width and the channel number of the characteristic diagram to obtain a characteristic matrix;

and the static feature extraction unit is used for vectorizing the feature matrix into the static features of the static sample.

10. The wildfire detection system as claimed in claim 9, wherein the dynamic feature extraction module specifically comprises:

the motion region extraction unit is used for extracting a motion region in the dynamic sample by using a single Gaussian background model method;

the flame texture feature extraction unit is used for extracting flame texture features from the motion area by utilizing a gray level co-occurrence matrix;

the area change characteristic and flicker characteristic extraction unit is used for extracting the area change characteristic and flicker characteristic of the motion area;

and the dynamic characteristic extraction unit is used for weighting and combining the flame texture characteristic, the area change characteristic and the flicker characteristic into the dynamic characteristic of the dynamic sample.

Images