CN116862894A - An industrial defect detection method based on image restoration - Google Patents

Abstract

The invention discloses an industrial defect detection method based on image recovery, which belongs to the technical field of surface defect detection of industrial products. The present invention uses the FFM-SA module to effectively fuse the features output by the encoder with the multi-scale features that filter out abnormalities through the block pyramid memory module, thereby enhancing the detailed information of the restored image; in addition, the segmentation sub-network proposed by the present invention extracts abnormal images and Recover the multi-scale features of the image and complete the cascade operation of the corresponding scale features to provide more effective information for locating abnormal areas. Finally, experiments were conducted using representative anomaly detection data sets, which proved that the method of the present invention has better detection results than other anomaly detection methods.

Description

An industrial defect detection method based on image restoration

Technical field

The invention relates to an industrial defect detection method based on image recovery, and belongs to the technical field of surface defect detection of industrial products.

Background technique

In the actual production process, due to the influence of various factors, defects will inevitably exist on the surface of industrial products, and these defects will have a negative impact on the quality and performance of the product. Therefore, in order to ensure product quality and enhance product competitiveness, surface defect detection has become crucial.

Compared with general target detection tasks, surface defect detection faces more difficulties, such as few defect samples, low visibility of defects, irregular shapes, and unknown defect types. In recent years, various surface defect detection algorithms based on unsupervised learning have been proposed. These algorithms are mainly divided into feature-based methods and reconstruction-based methods.

Feature-based methods detect anomalies by projecting the input image into a distinguishable feature space through a pre-trained model. This type of method usually has better performance, but poor interpretability and adjustability.

Reconstruction-based methods assume that normal samples can be reconstructed better through the latent feature space than abnormal samples. After the reconstruction network is trained on normal images, it fully learns the characteristics of normal images, so that it can reconstruct normal images well during inference; for abnormal images, because no abnormal features are learned during the training stage, the reasoning The ability to reconstruct anomalies is poor; finally, anomaly detection is performed by calculating the reconstruction error between the input image and the reconstructed image.

Reconstruction-based methods can directly observe the differences between the original image and the reconstructed image, making it easier to understand. However, in practical applications, due to the generalization of convolutional neural networks, reconstruction-based methods can sometimes reconstruct anomalies well, making it difficult to detect anomalies through reconstruction errors, resulting in a decrease in defect detection accuracy.

Contents of the invention

In order to improve the accuracy of the reconstruction-based defect detection method, the present invention provides an industrial defect detection method based on image restoration, including:

Step 1: Obtain the original image of the workpiece to be tested, input it into the encoder, and obtain a set of multi-scale feature maps;

Step 2: Use the block pyramid memory module to suppress abnormal features of the deep feature maps to obtain a set of abnormally filtered feature maps; the block pyramid memory module uses normal images for training;

Step 3: Use the feature fusion module (FFM-SA) based on the spatial attention mechanism to fuse the multi-scale feature map obtained in the step 1 and the anomaly-filtered feature map obtained in the step 2 to obtain enhanced detail information. feature map;

Step 4: Input the feature map of enhanced detail information obtained in Step 3 into the decoder to perform image restoration;

Step 5: Input the original image and the restored image obtained in step 4 into the segmentation sub-network to obtain the defect segmentation result, thereby determining the defects in the workpiece to be tested.

Optionally, the calculation process of the block pyramid memory module in step 2 includes:

For the original image x, the encoder E outputs a set of multi-scale feature maps {Z ₁ , Z ₂ ,..., Z ₅ }, and inputs the deep feature maps {Z ₃ , Z ₄ , Z ₅ } into the corresponding Block Pyramid Memory Module;

The third layer feature map The processing includes:

(1) In the first branch, the feature map Z ₃ is equally divided into blocks, where/> and/> are the division ratios along the height, width, and channel dimensions respectively; the height of each block is/> The width is/> The channel length is/> H is the height, W is the width, and C is the channel dimension;

For each block, flatten it to get a vector as the query, according to the query and memory The attention weight w is calculated based on the similarity of the memory items in , where/> Memory M ₁ contains N ₁ real-valued vectors with dimensions P ₁ , P ₁ =h ₁ ×w ₁ ×c ₁ ;

Then linearly combine the memory items in memory M ₁ by using the weight w, and finally transform the obtained feature vector into a feature map with the same dimension size as Z ₃

(2) In the second branch, the division rate is r _H2 = 2r _H1 , r _W2 = 2r _W1 , r _C2 = 2r _C1 , and the memory is expressed as Other calculation processes are the same as in the first branch. The feature map is obtained through query and aggregation.

(3) Convert the enhanced feature map to After feature fusion, the final feature map is obtained/>

The processing process of the fourth layer feature map Z ₄ and the fifth layer feature map Z ₅ is the same as the third layer feature map Z ₃ , thus obtaining a set of transformed feature maps

Optional, the calculation process of the FFM-SA module includes:

The repaired feature map output by the block pyramid memory module and/> and/> Enter the FFM-SA module;

First of all and F _abnormal achieve feature fusion through element-by-element summation:

Next, use a K×K convolution kernel to perform a convolution operation on F _fuse to obtain a feature map with a channel dimension of 3, which is the attention feature map. and/>

The attention feature map is normalized in the channel dimension through the softmax function. The formula is as follows:

in and/> Respectively expressed/> and/> The i-th row and j-th column elements;

Finally, three standardized attention feature maps are used to select and F _abnormal information to obtain the final output feature map F _out . The calculation formula is as follows:

where ⊙ represents the dot product operation; thus the enhanced feature map corresponding to the third layer feature map Z ₃ is obtained;

For the fourth layer feature map Z ₄ and the fifth layer feature map Z ₅ , the same processing is performed respectively to obtain the corresponding enhanced feature map.

Optionally, step five includes:

Step 51: Use ResNet34 as the feature extraction network of the segmentation module, input the restored image and the original image into the network respectively, extract multi-scale feature maps respectively, and splice the two sets of feature maps in channel dimensions respectively;

Step 52: Perform multi-scale feature fusion on the concatenated feature maps;

The CA attention mechanism is used to obtain cross-channel information, direction perception and position information, and 1×1 convolution is used for feature fusion and channel dimensionality reduction;

Align the resolutions of feature maps at different scales through upsampling, then use convolution to align channel dimensions, and finally perform element-wise addition operations to achieve multi-scale feature fusion;

After each layer of feature maps undergoes multi-scale feature fusion, a 3×3 convolution operation is performed;

After the feature maps except the first layer are upsampled to the same resolution size as the first layer feature map, the channel dimensions are spliced and then input into the segmentation head for defect segmentation to obtain abnormal segmentation results.

Optionally, use the encoder and decoder of UNet as the encoder and decoder of the reconstructed subnetwork.

Optionally, in the process of training the overall network, forge abnormal samples based on Perlin noise and Bezier curve. The specific process includes:

Use the saliency detection method CPD to obtain the salient area of the normal image, then use Perlin noise and Bezier curve to generate a defect mask, combine the defect mask with the abnormal source image and paste it in the salient area of the normal image to obtain the forged Defective samples.

Optionally, the training process is divided into two stages;

The first stage trains the reconstruction network based on the block pyramid memory module using only normal samples;

The second stage uses the forged defect samples to train an anomaly detection and segmentation network based on image restoration, and no longer updates the weights of the block pyramid memory module.

Optional, the reconstruction loss in the first stage is:

in, is the L ₂ loss function,/> is the SSIM loss function.

Optional, the loss in the second stage is:

Loss＝L _rec2 +L _seg

Among them, L _rec2 is the recovery loss, and L _seg is the segmentation loss;

L _seg =Loss _focal (I _m ,AM)

Among them, I _m represents the ground truth, AM represents the segmentation result, and Loss _focal represents the Focal Loss function.

The present invention also provides a computer-readable storage medium that stores computer-executable instructions. When the computer-executable instructions are executed by a processor, any one of the above methods is implemented.

The beneficial effects of the present invention are:

Through the following description of the technical features of the present invention, the disadvantage that abnormal images may be reconstructed well when conventional reconstruction methods are used and therefore cannot detect abnormalities through reconstruction errors is solved, thereby achieving the purpose of accurately detecting surface defects in industrial products.

1) A fake anomaly method is proposed to generate defect samples. By improving the previous forgery anomaly methods, a significant anomaly forgery method (SAFM-PB) based on Perlin noise and Bezier curve is proposed. Generate defects in the salient foreground area to avoid the influence of noise in the background area; use Perlin noise and Bezier curves to generate various defect shapes, and set the abnormal source image to an image different from the distribution of the data set or randomly sampled Normal images, increase the diversity of defects and encourage the model to learn the irregularities of abnormal samples.

2) In order to enhance the detailed information of the recovered image, a feature fusion module (FFM-SA) based on the spatial attention mechanism is proposed to effectively fuse the features extracted by the encoder and the abnormal multi-scale features filtered out by the block pyramid memory module. Enhance the detailed information of the restored image and improve the accuracy of anomaly detection.

3) Add a segmentation sub-network to extract multi-scale features of abnormal images and restored images, and complete cascade operations of corresponding scale features to provide more effective information for locating abnormal areas.

Description of the drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without exerting creative efforts.

Figure 1 is an overall network structure diagram of an industrial defect detection method based on image recovery according to the present invention.

Figure 2 is a reconstruction network structure diagram based on the block pyramid memory module.

Figure 3 is an example of a forged strip-shaped or block-shaped defect mask and its corresponding defect sample.

Figure 4 is a network structure diagram of the feature fusion module (FFM-SA) based on the spatial attention mechanism.

Figure 5 is a network structure diagram of segmented sub-networks.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

Example 1:

This embodiment provides an industrial defect detection method based on image restoration, including:

Step 1: Obtain the original image of the workpiece to be tested, input it into the encoder, and obtain a set of multi-scale feature maps;

Step 2: Use the block pyramid memory module to suppress abnormal features of the deep feature maps to obtain a set of abnormally filtered feature maps; the block pyramid memory module uses normal images for training;

Step 3: Use the feature fusion module (FFM-SA) based on the spatial attention mechanism to fuse the multi-scale feature map obtained in the step 1 and the anomaly-filtered feature map obtained in the step 2 to obtain enhanced detail information. feature map;

Step 4: Input the feature map of enhanced detail information obtained in Step 3 into the decoder to perform image restoration;

Step 5: Input the original image and the restored image obtained in step 4 into the segmentation sub-network to obtain the defect segmentation result, thereby determining the defects in the workpiece to be tested.

Example 2:

This embodiment provides an industrial defect detection method based on image restoration. The overall network structure to implement the method is shown in Figure 1, including a restoration sub-network and a segmentation sub-network, where the restoration sub-network consists of an encoder E, a decoder D. It consists of the Salient Anomaly Forgery Module (SAFM-PB) based on Perlin noise and Bezier curve, the block pyramid memory module (M) and the feature fusion module (FFM-SA) based on the spatial attention mechanism, in which UNet's The encoder and decoder serve as the encoder E and decoder D of the recovery sub-network.

The process of network training includes two stages. The first stage uses normal samples to train the reconstruction network based on the block pyramid memory module shown in Figure 2, which is designed to store the general patterns of normal samples and filter out abnormal features for subsequent recovery tasks. In the second stage, the network shown in Figure 1 is trained, and the weights of the block pyramid memory module are no longer updated at this time. The following is a detailed introduction to the two-stage training process:

1. Training of reconstruction network based on block pyramid memory module

The first stage trains the reconstruction network based on the block pyramid memory module, using only normal samples for training. The network structure is shown in Figure 2, including the encoder E and the decoder D. By adding skip connections and Block pyramid memory module to filter abnormal features.

For the input image x, the encoder E outputs a set of features, denoted as {Z ₁ , Z ₂ ,..., Z ₅ }. Take the deep feature map {Z ₃ , Z ₄ , Z ₅ } as the input of its corresponding block pyramid memory module.

Here, this embodiment uses the third layer feature map output by the encoder E (H is height, W is width, C is channel dimension) As an example (shown in the lower dotted box in Figure 2), the storage and reading process of the block pyramid memory module is introduced in detail:

(1) In the first branch, the feature map Z ₃ is equally divided into blocks, where/> and/> are the division ratios along the height, width, and channel dimensions, respectively. The height of each block is/> The width is/> The channel length is/> For each block, flatten it to get a vector as the query, according to the query and memory/> (Indicates that memory M ₁ contains N ₁ real-valued vectors with dimensions P ₁ , P ₁ =h ₁ ×w ₁ ×c ₁ ) to calculate the attention weight w, where/> Then linearly combine the memory items in memory M ₁ by using the weight w (the process is shown in the right dotted box in Figure 2), and finally transform the obtained feature vector into a feature with the same dimension size as Z ₃ Picture/>

(2) The second branch is similar to the first branch, but the division rate is r _H2 = 2r _H1 , r _W2 = 2r _W1 , r _C2 = 2r _C1 , and the memory is expressed as The feature map is obtained after querying and aggregating/>

(3) Convert the enhanced feature map to After feature fusion (channel splicing, then 1×1 convolution dimensionality reduction), the final feature map/>

The same operation can be performed for Z ₄ and Z ₅ , thus obtaining a set of transformed feature maps It is then sent to the decoder D for reconstruction of the input image x.

2. Training of anomaly detection and segmentation network based on image restoration

The second stage trains the anomaly detection and segmentation network based on image restoration, as shown in Figure 1. For the input normal image x, the abnormal image x′ is obtained through the SAFM-PB module. After inputting to the encoder E, a set of multi-scale features is output. Use the block pyramid memory module to suppress abnormal feature output and obtain a set of abnormally filtered feature maps, which are then fused with the multi-scale features output by the encoder E through the FFM-SA module to obtain a set of feature maps with enhanced detail information, which are sent to decoding. Device D is used for image recovery. Finally, the forged abnormal images and restored images are input into the segmentation subnetwork for defect segmentation.

1. Significant anomaly forgery method based on Perlin noise and Bezier curve

This invention forges abnormal samples based on Perlin noise and Bezier curve. In previously proposed models, spurious anomalies can easily appear in the background. In order to make the abnormality generated in the significant foreground area instead of the background area, the present invention obtains the salient area of the normal image by using the saliency detection method CPD, and then uses Perlin noise and Bezier curve to generate a defect mask, and combines the defect mask with The abnormal source images are combined and pasted in the salient area of the normal image to obtain forged defect samples, making the generated abnormal samples more conducive to model learning.

This invention uses Perlin noise to generate a defect mask, combines it with the abnormal source image and then pastes it into the salient area of the normal image to obtain a defect sample. In addition, during the production process of products, strip defects such as scratches, cracks, and strip obstructions, and block defects such as color contamination are prone to occur. The present invention forges strip-shaped and block-shaped defect samples through Bezier curves, and randomly uses second-, third- or fourth-order Bezier curves to obtain strip-shaped defect masks. The block defect mask consists of three Bezier curves. Each curve is a random second, third or fourth order Bezier curve. The next curve is the end point of the previous curve. The three curves are connected to form a closed figure. Filling is performed to obtain a block defect mask. Finally, the defect mask is combined with the abnormal source image and pasted in the salient area of the normal image to obtain a forged defect sample. Figure 3(a) shows the forged strip defect mask and the corresponding defect sample, and Figure 3(b) shows the block defect mask and the corresponding defect sample.

2. Space-based attention mechanism feature fusion module

The present invention proposes the structure of the FFM-SA module, and its network structure is shown in Figure 4. This invention enhances the third layer feature map output by the encoder E Take an example to describe this module in detail.

The input to this module includes the inpainted feature map output by the block pyramid memory module. and/> and/> (i.e. F _abnormal ), where/> and/> Indicates the repaired feature maps obtained by storing and reading memory blocks with sizes h ₁ × w ₁ × c ₁ and h ₂ × w ₂ × c ₂ respectively.

First of all and F _abnormal achieve feature fusion through element-by-element summation:

Next, use K×K convolution kernel to perform convolution operation on F _fuse , assuming that the size of K should be larger than h ₂ or w ₂ (for example: assuming the sizes of the two memory blocks in the third layer are 2×2×c ₁ respectively , 4×4×c ₂ , then K=5), and obtain the feature map with channel dimension 3, that is, the attention feature map and/> Compared with the original FFM module, this invention only performs convolution operation on the feature map F _fuse through a K×K convolution kernel to obtain three attention maps, and the channel dimension of the attention map is only 1. After improvement, the It greatly reduces the calculation amount of the entire module, significantly reduces the complexity of the module, and makes the module more lightweight.

The attention feature map is normalized in the channel dimension through the softmax function. The formula is as follows:

in and/> Respectively expressed/> and/> The i-th row and j-th column elements.

Finally, three standardized attention feature maps are used to select and F _abnormal information to obtain the final output feature map F _out . The calculation formula is as follows:

where ⊙ represents the dot product operation.

A set of enhanced feature maps is thus obtained, which are then sent to the decoder D for restoring the input image x′.

3. Split subnetworks

The pixel difference between the input image and the restored image of the restored sub-network is often used for defect detection and segmentation, but a single pixel has no semantic information, and distinguishing normal features from abnormal features requires rich contextual information. In addition, the size of industrial defects in actual production is usually multi-scale. Therefore, contextual information and multi-scale information are required to achieve accurate defect detection and segmentation. Therefore, the present invention proposes a multi-scale feature fusion module, whose network structure is shown in Figure 5.

This invention uses ResNet34 as the feature extraction network of the segmentation module, inputs the restored image and the forged abnormal image into the network respectively, extracts multi-scale feature maps, and obtains the four-layer feature maps of the forged abnormal image as follows: and/> The four-layer feature maps of the restored image are respectively and/> These two sets of feature maps are spliced in channel dimensions to provide more effective information for locating abnormal areas.

Finally, multi-scale feature fusion is performed on the concatenated feature maps. First, cross-channel information, direction perception and position information are obtained through the CA attention mechanism, thereby helping the model to more accurately locate and identify targets of interest. Then use 1×1 convolution for feature fusion and channel dimensionality reduction, and then continue with multi-scale information fusion: first align the resolutions of feature maps of different scales through upsampling, and then use convolution to align the channel dimensions. Finally, element-by-element addition is performed to achieve multi-scale feature fusion. After each layer of feature maps undergoes multi-scale feature fusion, a 3×3 convolution operation is performed. After the second, third, and fourth layer feature maps are upsampled to the same resolution size as the first layer feature map, the channel dimensions are spliced and then input into the segmentation head for defect segmentation to obtain an abnormal segmentation result, recorded as AM. And do loss calculation with pixel-level labels.

4.Loss function

For the first stage, the model is trained by combining _L2 loss and block-based SSIM loss function, and the reconstruction loss is:

For the second stage, recovery losses and segmentation losses are included. Normal images are input into the model for training with forged abnormal samples obtained by the SAFM-PB method. However, they face a huge category imbalance during the training process. Therefore, the Focal Loss function is used as the loss function of the segmentation sub-network to alleviate the category imbalance problem. , by penalizing common categories and focusing on training difficult samples to improve the ability to accurately segment these samples, thereby enhancing the robustness of the model. The segmentation loss is:

L _seg =Loss _focal (I _m ,AM)

Among them, I _m represents the ground truth and AM represents the segmentation result.

The recovery loss in the second stage is:

The total loss function in the second stage is:

Loss＝L _rec2 +L _seg

5. Abnormality identification

The present invention uses segmentation anomaly maps to directly evaluate image-level anomaly scores, that is, to determine whether there are anomalies in the image. First, AM is smoothed through mean filter convolution to gather local anomaly information. Finally, the maximum value of the smoothed anomaly map is used as the final image-level anomaly score score:

score=max(AM*f _sf×sf )

Among them, sf×sf means that the size of the mean filter is sf×sf, * represents the convolution operation, and AM represents the segmentation result.

After the above two stages of training, the trained network structure can be used to detect industrial defects. The detection process is as follows:

Step 1: Obtain the original image of the workpiece to be tested, input it into the encoder, and obtain a set of multi-scale feature maps;

Step 2: Use the block pyramid memory module to suppress abnormal features of the deep feature maps to obtain a set of abnormally filtered feature maps; the block pyramid memory module uses normal images for training;

Step 3: Use the feature fusion module FFM-SA based on the spatial attention mechanism to fuse the multi-scale feature map obtained in step 1 and the abnormally filtered feature map obtained in step 2 to obtain a feature map with enhanced detail information;

Step 4: Input the feature map with enhanced detail information obtained in Step 3 into the decoder for image restoration;

Step 5: Input the original image and the restored image obtained in step 4 into the segmentation sub-network to obtain the defect segmentation result, thereby determining the defects in the workpiece to be tested. In order to further illustrate the beneficial effects of the present invention, experiments and analyzes were conducted as follows:

1. Dataset and experimental settings

The present invention uses the industrial anomaly detection data set MVTec AD, which is specially proposed for unsupervised industrial visual anomaly detection and is currently an important benchmark in this field. The data set includes 3629 normal images for training and 1725 images for testing (including normal images and abnormal images). It consists of 15 categories of objects, including 10 object categories and 5 texture categories. Each category The images include a training set and a test set. The resolution of the image ranges from 700 to 1024. The challenge of this data set is that the difference between some abnormal images and normal images is small, and the size of the abnormal area is quite different. During experiments, the image size was adjusted to 256×256.

The method of this invention is implemented using PyTorch 1.8.1 and CUDA 11.1, and all experiments are run on RTX 3090GPU. Using the Adam optimizer, the initial learning rate is set to lr=0.0001, the learning rate is multiplied by 0.2 after 0.8×num_Epoch and 0.9×num_Epoch, gamma=0.2; the first-stage training Epoch is 600, and the second-stage training Epoch is 400; batch_size= 4.

In the experiment, for the bottle, cable, metal_nut and transistor categories, set and/> Remaining category settings/> and/>

2. Performance evaluation indicators

This experiment uses AUROC (Area Under ROC Curve, which is the area under the ROC curve) to evaluate the performance of the model. When the AUROC score is higher, the model performance is better.

The test sample generates an anomaly score through the anomaly detection network. When the anomaly score is greater than a certain threshold, it is determined to be an abnormal sample. Normal samples are regarded as negative examples (Negative), abnormal samples are regarded as positive examples (Positive), correctly detected abnormal samples are regarded as true examples (TP), correctly detected normal samples are regarded as true negative examples (TN), and normal samples are regarded as true negative examples (TN). The wrong detection of abnormal samples is called false positives (FP), and the wrong detection of abnormal samples as normal samples is called false negatives (FN). Different thresholds will produce a series of true positive rates (TPR) and false positives. Case rate (FPR), the calculation formula is as follows:

Sort the test samples according to the predicted anomaly score, and predict the samples less than or equal to the threshold as positive examples one by one in this order. Calculate FPR and TPR. With FPR as the horizontal axis and TPR as the vertical axis, draw the ROC curve in the coordinate system. , the method of the present invention is evaluated by the area under the final curve AUROC.

3. Experimental comparison results and analysis

The method of the present invention is compared with the current classic anomaly detection algorithms. Table 1 shows the image-level anomaly detection results (the first 5 rows represent texture classes, the last 10 rows represent object classes, Avg.tex, Avg.obj and Avg.all represent the texture class, object class and the average AUROC score of all classes respectively).

Table 1 Comparison of AUROC scores between the present invention and other existing methods in image-level anomaly detection tasks

As can be seen from Table 1, the defect detection performance of the inventive method is better than many classic anomaly detection algorithms, 0.8 percentage points higher than the highest average AUROC score among these algorithms, and 8 categories out of 15 categories obtained the highest AUROC score, and achieved respectable AUROC scores in other categories as well. In the object class, the inventive method outperforms all other methods. In the texture class, the performance of the method of the present invention is slightly lower than CutPaste and higher than other methods. Compared with the CutPaste method, the advantage of the method of the present invention is that the average AUROC of the object category and all categories is significantly better than the CutPaste method.

The method of the present invention is compared with some mainstream anomaly detection algorithms on pixel-level anomaly detection tasks. As shown in table 2:

Table 2 Comparison of AUROC scores between the method of the present invention and existing methods in pixel-level anomaly detection tasks

It can be seen from Table 2 that the method of the present invention improves the accuracy of positioning and has better anomaly detection performance than other methods. The method of the present invention obtained the highest pixel-level average AUROC scores and mAP scores, and obtained the highest AUROC scores in 7 categories. In addition, the method of the present invention achieves optimal detection performance regardless of texture type or object type.

To sum up, in order to solve the problems of reconstruction blur and reconstruction anomalies existing in reconstruction-based methods, the present invention proposes a method of industrial anomaly detection and segmentation based on image restoration. By forging abnormal samples through the significant anomaly forgery method based on Perlin noise and Bezier curve, it increases the diversity of defective samples and encourages the model to learn the irregularities of abnormal samples. The FFM-SA module is used to effectively fuse the features output by the encoder with the multi-scale features filtered out by the block pyramid memory module to enhance the detailed information of the reconstructed image. The proposed segmentation sub-network extracts multi-scale features of abnormal images and restored images, and completes the cascade operation of corresponding scale features to provide more effective information for locating abnormal areas. Finally, experiments were conducted using representative anomaly detection data sets, which proved that the method of the present invention has better detection results than other anomaly detection algorithms.

Some steps in the embodiments of the present invention can be implemented using software, and corresponding software programs can be stored in readable storage media, such as optical disks or hard disks.

The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection of the present invention. within the range.

Claims (10)

1. An industrial defect detection method based on image restoration, the method comprising:

step one: acquiring an original image of a workpiece to be detected, and inputting the original image into an encoder to obtain a group of multi-scale feature images;

step two: the method comprises the steps of utilizing a block pyramid memory module to inhibit abnormal characteristics of a deep characteristic map, and obtaining a group of characteristic maps after filtering abnormality; the block pyramid memory module adopts normal images for training;

step three: the multi-scale feature map obtained in the first step and the feature map obtained in the second step after abnormal filtering are fused through a feature fusion module (FFM-SA) based on a spatial attention mechanism, so that a feature map of enhanced detail information is obtained;

step four: inputting the feature map of the enhanced detail information obtained in the third step into a decoder for image recovery;

step five: inputting the original image and the recovery image obtained in the step four into a segmentation sub-network to obtain a defect segmentation result, thereby judging the defects of the workpiece to be detected.

2. The method for detecting industrial defects based on image restoration according to claim 1, wherein the calculating process of the block pyramid memory module in the second step comprises:

for the original image x, the encoder E outputs a set of multi-scale feature maps { Z } ₁ ,Z ₂ ,...,Z ₅ Deep feature map { Z } ₃ ,Z ₄ ,Z ₅ Respectively inputting the corresponding block pyramid memory modules;

third layer of feature mapThe processing procedure of (1) comprises:

(1) In the first branch, the feature map Z ₃ Equally divided intoIndividual blocks, wherein-> and />Dividing rates along the height, width and channel dimensions, respectively; the height of each block is +.>Width is->Channel length is +.>H is high, W is wide, and C is the channel dimension;

for each block, flattening to obtain a vector as query, and according to the query and the memorySimilarity of the memory items in (a) to calculate the attention weight w, wherein +.>Memory M ₁ Comprising N ₁ With dimensions P ₁ Real value vector, P ₁ ＝h ₁ ×w ₁ ×c ₁ ；

Then, the weight w is used for the memory M ₁ Linear combination is carried out on the memory items in the memory, and finally the obtained characteristic vector is transformed into a vector with Z ₃ Feature maps of the same dimension size

(2) In the second branch, the division rate is r _H2 ＝2r _H1 ，r _W2 ＝2r _W1 ，r _C2 ＝2r _C1 The memory is expressed asOther calculation processes are the same as in the first branch, and the characteristic diagram is obtained after inquiring and aggregating>

(3) Enhanced feature mapAfter feature fusion, the final feature map is obtained

Fourth layer of feature map Z ₄ And fifth layer characteristic diagram Z ₅ And the third layer characteristic diagram Z ₃ Identical, thereby obtaining a set of transformed feature maps

3. The method for detecting an industrial defect based on image restoration according to claim 2, wherein the calculation process of the FFM-SA module includes:

outputting the repair feature map output by the block pyramid memory module and /> and />Inputting the FFM-SA module;

first to and F_abnormal Feature fusion is achieved by element-wise summation:

next, F is checked using a K convolution kernel _fuse Performing convolution operation to obtain a feature map with channel dimension of 3, namely a attention feature map and />

And normalizing the attention characteristic diagram in the channel dimension through a softmax function, wherein the formula is as follows:

wherein and />Respectively indicate-> and />I-th, j-th column element of (a);

finally, three normalized attention profile graphs are used to select and F_abnormal Information obtaining final output feature map F _out The calculation formula is as follows:

wherein "+.; thereby obtaining a third layer characteristic diagram Z ₃ Corresponding enhanced feature graphs;

for the fourth layer of feature map Z ₄ And fifth layer characteristic diagram Z ₅ And respectively carrying out the same processing to obtain corresponding enhanced characteristic diagrams.

4. The method for detecting industrial defects based on image restoration according to claim 3, wherein the fifth step comprises:

step 51: the ResNet34 is used as a characteristic extraction network of the segmentation module, a restored image and an original image are respectively input into the network, multi-scale characteristic images are respectively extracted, and channel dimension splicing is respectively carried out on the two groups of characteristic images;

step 52: carrying out multi-scale feature fusion on the feature graphs after cascading;

acquiring cross-channel information, direction sensing and position information through a CA attention mechanism, performing feature fusion by using 1X 1 convolution, and performing channel dimension reduction;

the resolutions of feature graphs with different scales are aligned through up-sampling, then the channel dimensions are aligned through convolution, and finally element-by-element addition operation is performed, so that multi-scale feature fusion is realized;

after the multi-scale feature fusion is carried out on each layer of feature map, a convolution operation of 3 multiplied by 3 is carried out;

and upsampling the feature images except the first layer to the same resolution as the feature images of the first layer, splicing the channel dimensions, and inputting the spliced feature images into a segmentation head for defect segmentation to obtain an abnormal segmentation result.

5. The method for detecting industrial defects based on image restoration according to claim 1, wherein an encoder and a decoder of UNet are used as the encoder and the decoder of the restoration sub-network.

6. The method for detecting industrial defects based on image restoration according to claim 1, wherein the training of the network is based on Perlin noise and bezier curve falsification of abnormal samples, and the specific process comprises:

and obtaining a salient region of the normal image by using a salient detection method CPD, generating a defect mask by using Perlin noise and Bezier curve, combining the defect mask with the abnormal source image, and pasting the combined defect mask with the abnormal source image in the salient region of the normal image to obtain a forged defect sample.

7. The method for detecting industrial defects based on image restoration according to claim 6, wherein the training process is divided into two stages;

in the first stage, training a reconstruction network based on a block pyramid memory module by using normal samples only;

the second stage uses the forged defect sample to train an anomaly detection and segmentation network based on image restoration, and does not update the weights of the block pyramid memory module.

8. The method for detecting an industrial defect based on image restoration according to claim 7, wherein the reconstruction loss in the first stage is:

wherein ,is L ₂ Loss function (F)>Is an SSIM loss function.

9. The method for detecting an industrial defect based on image restoration according to claim 7, wherein the loss in the second stage is:

Loss＝L _rec2 +L _seg

wherein ,L_rec2 To recover loss, L _seg Loss for segmentation;

L _seg ＝Loss _focal (I _m ,AM)

wherein ,I_m Represents the group trunk, AM represents the segmentation result, loss _focal Representing the Focal Loss function.

10. A computer readable storage medium storing computer executable instructions which when executed by a processor implement the method of any one of claims 1 to 9.