CN118396947A - Steel micro defect detection method suitable for edge equipment deployment - Google Patents

Abstract

The invention discloses a steel micro defect detection method suitable for edge equipment deployment, which relates to the field of steel micro defect identification of industrial environments, and is characterized in that a steel defect image is compressed and processed into an image file with a specified size, the image is input into a characteristic extraction network based on a U-shaped structure, a deformable convolution and attention module of the characteristic extraction network is responsible for extracting characteristics, a point product mode of an attention mechanism acquires context information characteristics in the steel defect, meanwhile, in order to keep certain initial characteristics of the steel defect, fine granularity fusion is carried out on the extracted local significant characteristics and global context information characteristics to obtain final output characteristics, the final output characteristics are transmitted to a decoder part, the characteristic mapping is carried out by the decoder, and the steel micro defect detection result is finally output, so that the characteristic extraction network is conveniently deployed at an edge equipment end, a knowledge distillation learning frame is provided, and an objective function is continuously optimized through learning of a teacher network, so that the guidance of a student network is achieved.

Description

A method for detecting tiny defects in steel suitable for edge device deployment

Technical Field

The present invention relates to the field of steel micro-defect recognition in industrial environments, and in particular to a steel micro-defect target detection method suitable for edge device deployment.

Background technique

With the rapid development of science and technology, productivity has made a qualitative leap. The world's steel industry will face new opportunities and challenges. Steel is an important material basis for national construction and the realization of the four modernizations. Especially in the automotive, construction, aerospace, electronic machinery and other industries, the demand for steel production is growing. The generation of surface defects in steel plates directly affects the production efficiency of enterprises. Steel defect target detection aims to find all defect targets of interest in the production of steel through detection algorithms, and determine their locations and categories; Existing steel defect detection methods can generally be divided into three categories: methods based on manual visual inspection, methods based on eddy current detection, and methods based on machine vision; the method based on manual visual inspection is the most primitive way to detect steel surface defects. Although this method has high defect recognition accuracy, it requires a lot of manpower investment. The method based on manual visual inspection is time-consuming and labor-intensive, and its ability to undertake real-time detection is slightly insufficient; the method based on eddy current detection is used frequently in steel surface defect detection systems The method itself is easily affected by environmental noise, probe lift-off and changes in equipment structure. The feedback signal of the defect position of steel is often deteriorated. Therefore, before using this detection method, it is necessary to introduce signal processing technology to improve the signal-to-noise ratio of the defect position feedback signal, which also increases the technical difficulty of steel surface defect detection. Among the steel surface defect detection methods, the machine vision-based method, especially the deep learning-based defect detection method, has the best effect. The deep learning-based method extracts the image features of steel surface defects through convolutional neural networks, and through a series of operations, the defects on the steel surface can be accurately identified. However, the neural network model designed by the deep learning method is limited by the large number of network parameters, large amount of calculation and overly complex structure, and it is difficult to be actually deployed and applied on edge devices in industrial environments. In order to alleviate the above problems, this patent proposes a steel micro-defect detection method (A) suitable for edge device deployment. The steel micro-defect detection method suitable for edge device deployment) mainly includes three parts: a feature extraction module based on deformable convolution and attention, a feature extraction network based on U-shaped structure and a learning architecture based on knowledge distillation; the feature extraction module based on deformable convolution and attention uses deformable convolution to dynamically adjust the weights in the receptive field, flexibly captures the features in the image at a specific position, and uses the attention mechanism to make the module pay more attention to the area related to steel defects; the feature extraction network based on the U-shaped structure consists of an encoder and a decoder. The encoder consists of multiple feature extraction modules of deformable convolution and attention, and its purpose is to extract semantic features of various scales in the input image while retaining spatial information and local details. The decoder part is responsible for decoding the feature map extracted by the encoder into the final output; the learning architecture based on knowledge distillation performs knowledge distillation on the feature extraction network to obtain a more lightweight network model for deployment on the edge device.

Summary of the invention

The purpose of the present invention is to provide a method for detecting tiny defects in steel suitable for edge device deployment to solve the above-mentioned problems.

The present invention is achieved through the following technical solutions:

A method for detecting tiny defects in steel suitable for edge device deployment mainly includes the following steps:

S1: Use a professional camera to capture the actual image of steel, thereby obtaining the required raw image dataset of steel defects. By compressing the raw image dataset, the image is processed into a specified size, and the location of the steel defects is marked.

S2: Input the processed steel defect image dataset into the convolution component to extract the shallow features F of the steel surface defects, input the shallow features F into the offset weight learning pyramid composed of convolution blocks, and adjust the offset (ΔP _x ,ΔP _y ) of the deformable convolution by learning the offset weight matrix W;

(ΔP _x ,ΔP _y )＝W(F)

Where (ΔP _x ,ΔP _y ) is the offset of the deformable convolution, W is the output of the offset weight learning pyramid, and F is the shallow feature of the input.

S3: Use the learned offset (ΔP _x , ΔP _y ) to obtain the position p _l (x _l , y _l ) of the new sampled pixel feature value l located at the irregular position of the steel defect;

p _l (x _l ,y _l )=p _g ((x+Δp _x ),(y+Δp _y ))

Where l represents the newly sampled pixel value, p _l (x _l ,y _l ) is the position of the pixel value l, p _g is the previous sampled pixel value, and p _g (x, y) is the position of the pixel value p _g .

S4: The feature _F1 obtained by deformable convolution introduces an offset without considering the information interaction between adjacent pixels. To this end, the attention mechanism is used to interact with adjacent pixels. The pixel in the feature _F1 is recorded as _pq (x,y), and the dot product of _pq and _pk∈K is taken to calculate the correlation between a specific reference pixel and all sampled pixels.

Where Atten _qi is used to measure the relevance weight of a specific reference pixel to the i-th sampling pixel, K is the set of sampling offsets, k is the number of the sampling pixel, and p _k is the position of any pixel.

S5: In order to selectively aggregate the contextual information in steel defects and retain more semantic information in the global view, the contextual information in steel defects is extracted by adding all sampled pixels p _a with the corresponding attention weights. At the same time, in order to retain some initial features of steel defects, the extracted contextual information is fine-grainedly fused and output with the original reference pixels;

Where _pF represents the final output feature of the feature extraction module of deformable convolution and attention, p _a represents all sampled pixels, and _F1 represents some initial features of steel defects.

S6: The features finally output by the deformable convolution and attention feature extraction module are input into the decoder for feature mapping and the final result of steel micro-target detection is obtained.

S7: Finally, in order to be suitable for deployment on edge devices, the distillation learning architecture composed of the teacher network model and the student network model is used for knowledge distillation. The teacher network is continuously trained to optimize the loss function L _g , thereby achieving optimization guidance for the student network loss function L _st ;

_Lst ＝ _Lobj + _Lcla + _Lbox

Where K is the length and width of the steel defect detection box generated by the teacher network, B is the number of detection boxes, M _q is the number of detection boxes within the threshold range, s _ijc is the value of the detection box position generated by the student network model, t _ijc is the value of the detection box position generated by the teacher network model, L _obj represents the loss between the true box confidence and the confidence of the student model output network, L _cla represents the loss between the true classification probability and the category of the student model output network, and L _box represents the loss between the true position of the detected steel defect and the position of the steel defect output by the student model.

Compared with the prior art, the present invention has the following advantages and beneficial effects:

This patent is a method for detecting tiny defects in steel suitable for edge device deployment. Currently, the most commonly used steel defect detection methods are generally based on deep learning neural networks, which use neural networks to extract and detect steel defect features. However, existing neural networks have the characteristics of large number of parameters and large amount of calculation, and cannot be effectively deployed on edge devices. At the same time, the feature extraction of neural networks is limited by the receptive field of the convolution kernel, resulting in insufficient attention to tiny target defects in steel. In order to improve the accuracy of target detection of tiny defects in steel and make deep learning network models easy to deploy on edge devices, we propose a method for detecting tiny defects in steel suitable for edge device deployment to effectively extract local significant features and global information features of tiny defects in steel. This method compresses the steel defect image into an image of a specified size. The image is input into a feature extraction network based on a U-shaped structure. The deformable convolution and attention modules of the feature extraction network are responsible for extracting features. The deformable convolution performs irregular convolution on the input steel defect image to extract local significant features. The attention mechanism uses the dot product method to obtain the context information features in the steel defects. At the same time, in order to retain some initial features of the steel defects, the extracted local significant features are fine-grainedly fused with the global context information features to obtain the final output features. The final output features are passed to the decoder part, which performs feature mapping and finally outputs the results of steel micro-defect detection. Finally, in order to facilitate the deployment of the feature extraction network on the edge device, a knowledge distillation learning framework is proposed. Through the learning of the teacher network, the objective function is continuously optimized to guide the student network.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

FIG1 is a flow chart of a method for detecting minor defects in steel suitable for edge device deployment;

Figure 2 Deformable and attention feature fusion extraction module;

Fig. 3 Feature extraction network based on U-shaped structure;

Fig. 4 Distillation learning framework.

Detailed ways

In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and drawings. The schematic implementation modes and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention. It should be noted that the present invention is already in the actual development and use stage.

Example 1

As shown in FIG. 1 to FIG. 4 , this embodiment is a method for detecting minor steel defects suitable for edge device deployment:

S1: Get the image to be detected. The specific implementation steps are as follows:

(a) Use a professional camera to capture the actual image of the scene, thereby obtaining the required raw image dataset of steel micro-defects;

(b) compressing the image to a specified size and using it as a network input image;

(c) Annotate the compressed steel micro-defect image dataset and use it as a label for subsequent training;

A professional camera is used to capture actual images of steel, thereby obtaining the required raw image dataset of steel defects. The raw image dataset is compressed and processed into a specified size. The locations of steel defects are marked to facilitate subsequent network model training.

S2: The feature extraction module of deformable convolution and attention performs feature extraction. The specific implementation steps are as follows:

(a) The compressed steel micro-defect image dataset is input into the first layer of convolutional components to extract the shallow features F of steel surface defects:

F = ConvBlock(X)

Where X represents the input of steel micro-defect image dataset;

(b) The processed steel defect image dataset is input into the convolution component to extract the shallow features F of the steel surface defects. The shallow features F are input into the offset weight learning pyramid composed of convolution blocks, and the offset (ΔP _x , ΔP _y ) of the deformable convolution is adjusted by learning the offset weight matrix W. This process can be expressed by formula (2):

(ΔP _x ,ΔP _y )＝W(F)

Where (ΔP _x ,ΔP _y ) is the offset of the deformable convolution, W is the output of the offset weight learning pyramid, and F is the shallow feature of the input.

(c) Use the learned offset (ΔP _x , ΔP _y ) to obtain the position p _l (x _l , y _l ) of the new sampled pixel feature value l located at the irregular position of the steel defect

p _l (x _l ,y _l )=p _g ((x+Δp _x ),(y+Δp _y ))

Where l represents the newly sampled pixel value, p _l (x _l ,y _l ) is the position of the pixel value l, p _g is the previous sampled pixel value, and p _g (x, y) is the position of the pixel value p _g .

(d) The feature vector finally obtained by deformable convolution is F ₁ . The acquisition process of F ₁ is expressed as formula (5):

F ₁ =D _conv (O,W)

Among them, D represents the deformable convolution, O represents the offset learned by the deformable convolution, and W represents the deformable convolution learned.

The weight learned;

(e) Deformable convolution introduces spatial offsets without considering the information interaction between adjacent pixels. To this end, we introduce an attention mechanism to interact with adjacent pixels. The feature _F1 obtained by the deformable convolution is converted into two vectors, namely the key (K) and value (V) of the attention mechanism. At the same time, the pixels in the feature _F1 are recorded as _pq (x,y). The dot product of _pq and _pk∈K is taken to calculate the correlation between a specific reference pixel and all sampled pixels. The attention weight _Attenqk is obtained as shown in formula (6):

Where Atten _qi is used to measure the relevance weight of a specific reference pixel to the i-th sampling pixel, K is the set of sampling offsets, k is the number of the sampling pixel, and p _k is the position of any pixel.

(f) Finally, in order to selectively aggregate the contextual information in the tiny defects of steel and retain more semantic information in the global view, the contextual information in the tiny defects of steel is extracted by adding all the sampled pixels p _a with the corresponding attention weights. At the same time, in order to retain some initial features of the tiny defects of steel, the extracted contextual information is fine-grainedly fused with the original reference pixels. The process is shown in formula (7):

Where p _F represents the final output feature of the feature extraction module of deformable convolution and attention, p _a represents all sampled pixels, and F ₁ represents some initial features of steel defects;

S3: The feature extraction network extraction process based on the U-shaped structure, the specific implementation steps are as follows:

(a) The compressed steel image is input into the first convolutional block in the network to obtain the preliminary shallow features F′;

(b) The preliminary shallow features F′ are sequentially input into multiple deformable convolution and attention feature extraction modules to extract local features and global context features F″;

(c) The extracted local features and global context features F″ are sequentially input into multiple decoder structures to map and decode the features, and finally the results of steel micro-defect detection are output through a linear layer;

The processed steel image dataset is input into the U-shaped feature extraction network. The global spatial feature information of the steel image is integrated through the convolution layer and the mixed pooling layer. Then, the local feature extraction of the tiny defects of the steel is completed by learning the offset of the deformable convolution, and the contextual information features around the steel defects are obtained through the attention mechanism. In order to prevent the deformable convolution and attention mechanism modules from losing some original important features, the extracted local significant features are fine-grainedly fused with the global contextual information features around the steel defects to obtain the final output features. Finally, the decoder is used to map and decode the final output features, and the final results of steel tiny defect detection are obtained through decoding.

S4: Based on the knowledge distillation learning architecture, the feature extraction network is subjected to knowledge distillation. The specific implementation steps are as follows:

(a) The images of the steel defect dataset are input into the teacher model and the student model for training respectively;

(b) During the training process of the teacher network, the loss function L _g is generated because of the need to supervise and compare the student network. The teacher network needs to optimize the loss function L _g to continuously improve the guidance of the student network. The loss function L _g is shown in formula (8):

Where K is the length and width of the steel defect detection box generated by the teacher network, B is the number of detection boxes, M _q is the number of detection boxes within the threshold range, s _ijc is the value of the detection box position generated by the student network model, and t _ijc is the value of the detection box position generated by the teacher network model;

(c) The student network model is a feature extraction network with a U-shaped structure. During the student network training process, there will be a loss between the predicted value of the steel defect detection frame and the real data value of the detection frame, which is recorded as _Lst . _Lst consists of three parts, as shown in formula (9):

_Lst ＝ _Lobj + _Lcla + _Lbox

Where L _obj represents the loss between the true box confidence and the confidence of the student model output network, L _cla represents the loss between the true classification probability and the category of the student model output network, and L _box represents the loss between the true position of the detected steel defect and the position of the steel defect output by the student model;

(d) L _obj , L _cla and L _box are specifically expressed as shown in formulas (10), (11) and (12):

Where K represents the length and width of the input image, B represents the number of detection boxes, Indicates that there are steel defects in the detection frame. It means that there are no steel defects in the detection frame. represents the confidence of the real box, Represents the confidence of the predicted box, W _box represents the weight of the position regression loss, and L _MAE represents the mean absolute error between the true box and the predicted box position;

(e) In order for the teacher network in the knowledge distillation learning architecture to better guide the student network to learn, it is necessary to perform weighted operations on the loss function of the teacher network and the loss function of the student network:

L _f =μL _st +βL _g (μ + β = 1)

Where _Lf represents the loss function that the distillation framework finally optimizes, μ and β represent the weight values of the loss between the student model and the teacher model respectively;

The knowledge distillation learning framework is used to perform distillation learning on the U-based feature extraction network model, which is convenient for deployment on the device side. By continuously optimizing the objective function of the teacher network, the teacher network can learn more knowledge about the characteristics of small defects in dynamic steel. At the same time, knowledge distillation is performed on the teacher network to guide the optimization of the student network objective function.

The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims (1)

1. A method for detecting minor defects in steel suitable for edge device deployment, which mainly includes the following steps: S1: Use a professional camera to capture the actual image of steel, thereby obtaining the required raw image dataset of steel defects. By compressing the raw image dataset, the image is processed into a specified size, and the location of the steel defects is marked. S2: Input the processed steel defect image dataset into the convolution component to extract the shallow features F of the steel surface defects. Input the shallow features F into the offset weight learning pyramid composed of convolution blocks to adjust the offset of the deformable convolution by learning the offset weight matrix W. Where (ΔP _x ,ΔP _y ) is the offset of the deformable convolution, W is the output of the offset weight learning pyramid, and F is the shallow feature of the input. S3: Use the learned offset (ΔP _x , ΔP _y ) to obtain the position p _l (x _l , y _l ) of the new sampled pixel feature value l located at the irregular position of the steel defect; p _l (x _l ,y _l )=p _g ((x+Δp _x ),(y+Δp _y )) Where l represents the newly sampled pixel value, p _l (x _l ,y _l ) is the position of the pixel value l, p _g is the previous sampled pixel value, and p _g (x, y) is the position of the pixel value p _g . S4: The feature _F1 obtained by deformable convolution introduces an offset without considering the information interaction between adjacent pixels. To this end, the attention mechanism is used to interact with adjacent pixels. The pixel in the feature _F1 is recorded as _pq (x,y), and the dot product of _pq and _pk∈K is taken to calculate the correlation between a specific reference pixel and all sampled pixels. Where Atten _qi is used to measure the relevance weight of a specific reference pixel to the i-th sampling pixel, K is the set of sampling offsets, k is the number of the sampling pixel, and p _k is the position of any pixel. S5: In order to selectively aggregate the contextual information in steel defects and retain more semantic information in the global view, the contextual information in steel defects is extracted by adding all sampled pixels p _a with the corresponding attention weights. At the same time, in order to retain some initial features of steel defects, the extracted contextual information is fine-grainedly fused and output with the original reference pixels; Where _pF represents the final output feature of the feature extraction module of deformable convolution and attention, p _a represents all sampled pixels, and _F1 represents some initial features of steel defects. S6: The features finally output by the deformable convolution and attention feature extraction module are input into the decoder for feature mapping and the final result of steel micro-target detection is obtained. S7: Finally, in order to be suitable for deployment on edge devices, the distillation learning architecture composed of the teacher network model and the student network model is used for knowledge distillation. The teacher network is continuously trained to optimize the loss function L _g , thereby achieving optimization guidance for the student network loss function L _st ; _Lst ＝ _Lobj + _Lcla + _Lbox Where K is the length and width of the steel defect detection box generated by the teacher network, B is the number of detection boxes, M _q is the number of detection boxes within the threshold range, s _ijc is the value of the detection box position generated by the student network model, t _ijc is the value of the detection box position generated by the teacher network model, L _obj represents the loss between the true box confidence and the confidence of the student model output network, L _cla represents the loss between the true classification probability and the category of the student model output network, and L _box represents the loss between the true position of the detected steel defect and the position of the steel defect output by the student model.