CN109886970A - Detection and segmentation method of target objects in terahertz images and computer storage medium - Google Patents

Abstract

The invention discloses a detection and segmentation method of a target object in a terahertz image and a computer storage medium. The conditional generation confrontation network is used to organically fuse the two tasks of detection and segmentation, and the segmentation mask is generated for samples containing weapons. For samples that do not contain weapons, generate a picture with zero values of all pixels, set an appropriate loss function and an efficient network structure, when the training set is small and the image signal-to-noise ratio is low, the trained image The model can detect whether the input terahertz image contains the target object and segment the target object from the image. The invention can train an accurate detection and segmentation model under the condition that the number of training samples is small and the signal-to-noise ratio of the picture is very low, and the model has fast processing speed and high timeliness, and can be applied to In the real-time security inspection system, the labor intensity of manual security inspection is reduced and the efficiency is improved.

Description

Detection and segmentation method of target objects in terahertz images and computer storage medium

technical field

The invention relates to a method for detecting and segmenting a target object and a computer storage medium, in particular to a method for detecting and segmenting a target object in a terahertz image and a computer storage medium.

Background technique

Detecting objects hidden under clothing is a critical task in security checks. However, the current manual security inspection methods are very inefficient and have a high rate of missed inspections. Terahertz imaging technology provides a non-contact solution. Terahertz is an electromagnetic wave with a wavelength between 0.1×10 ¹² -10×10 ¹² Hz, which obtains images of objects hidden under clothing by measuring the radiant heat of different parts of the human body. And compared to X-rays, such electromagnetic waves are harmless to the human body.

In the case of limited labeled data, we not only need to determine whether the image contains hidden objects, but also need to accurately segment the detected objects. However, due to the physical factors inherent in terahertz imaging, the contrast and signal-to-noise ratio of the images are often low. The difference between the hidden object and the human body part in the pixel value is not obvious, and the edge of the object is almost mixed with the human body and cannot be distinguished. At the same time, there is a lot of background noise in the picture. Therefore, under such bad conditions, the algorithms leading to standard object segmentation and saliency detection do not have a good solution to the problem. Earlier work focused on statistical models, Shen (Shen, X., Dietlein, C., Grossman, E.N., Popovic, Z., Meyer, F.G.: Detection and segmentation of concealed objects in terahertz images. IEEE Transactions on Image Processing 17 (2008) ) 2465-2475) propose a multi-level threshold segmentation algorithm for modeling radiant temperature by a Gaussian mixture model. First, the anisotropic diffusion algorithm is used to remove noise, and then the boundary that changes with the threshold is traced to complete the segmentation task. LEE (Lee, D., Yeom, S., Son, J., Kim, S.: Automatic image segmentation for concealed object detection using the expectation-maximization algorithm. Optics Express 18 (2010) 10659-10667) also uses a Gaussian mixture model . The human body and the objects contained in the human body are segmented by obtaining the Bayesian boundary through an expectation-maximization algorithm twice. Similarly, Yeom (Lee, D.S., Son, J.Y., Jung, M.K., Jung, S.W., Lee, S.J., Yeom, S., Jang, Y.S.: Real-time outdoor concealed-object detection with passive millimeterwave imaging. Optics Express 19 (2011) 2530-2536) used a multi-scale segmentation algorithm based on a Gaussian mixture model, and at the same time, to achieve real-time performance, a vector quantization technique was used before the expectation maximization algorithm. However, these traditional algorithms do not have the ability to detect, that is, they all perform segmentation operations on the premise that the current image contains objects, and at the same time, under such poor imaging conditions, there is no good segmentation result.

Current instance segmentation algorithms based on deep convolutional neural networks have good potential to address this challenge. This type of method can be roughly divided into two categories, one is the method based on R-CNN recommendation. They use a bottom-up pipeline, where the segmentation results rely on the R-CNN recommendation regions, and then go to classify these results. Another class of methods is based on semantic segmentation. The implementation of instance segmentation by these methods depends on the results of semantic segmentation, that is, the results of semantic segmentation are obtained first, and then the pixels are divided into different instances. A state-of-the-art instance segmentation algorithm Mask-RCNN (He, K., Gkioxari, G., Dollar, P., Girshick, R.B.: Mask r-cnn. International Conference on ComputerVision (2017) 2980-2988) shared convolutional layers features, which can complete the tasks of detection, classification and segmentation at the same time. However, these methods generally treat segmentation and detection as two independent units, resulting in complex model structure, long processing time, and unrobustness. Moreover, this type of method requires a large amount of data to train a usable model. Obviously, this type of method is not suitable for target detection and segmentation tasks in terahertz scenes.

SUMMARY OF THE INVENTION

Purpose of the invention: The technical problem to be solved by the present invention is to provide a detection and segmentation method and a computer storage medium for a target object in a terahertz image, which overcomes the need for a large contrast between the target object and the background in the prior art or the need to collect and label a large number of high-resolution images. Defects such as quality samples, in the case where the signal-to-noise ratio is very low, the target object and the background are extremely similar in pixel density, and the number of training samples is small, accurate detection and segmentation results are obtained, and at the same time, it has high processing speed, enabling it to Applied to the real-time security inspection system.

Technical solution: The method for detecting and segmenting a target object in a terahertz image according to the present invention includes the following steps:

(1) Using the original terahertz image x in the training set as a training sample, manually mark the target object area in x to obtain the real segmentation mask y, and preprocess x and y at the same time, all the The pixel value is converted to between -1 and 1;

(2) Input the original terahertz image x and random noise z into the generator, and the generator generates a false segmentation mask;

(3) The real segmentation mask and the corresponding original terahertz image are formed into a real sample pair, and the false segmentation mask and the corresponding original terahertz image are formed into a fake sample pair, which are jointly input into the discriminator for judgment;

(4) Calculate the loss function of the generator and the discriminator separately, and use the gradient descent algorithm to optimize the generator to make the loss function of the generator as small as possible. It may be large until the model converges, return to step (1), and enter step (5) after traversing the training set;

(5) Go back to step (1) until the set number of iterations is reached;

(6) Save the model trained in the above steps, input the new image to be detected into the generator, and obtain the corresponding detection and segmentation results.

In order to keep the target area while discarding other useless details and obtain a simple and efficient network structure, some layers of the encoder and decoder can be selectively connected. The generator is a two-dimensional neural network G, which includes 8 convolutional layers and 8 transposed convolutional layers, in which the filter size and moving step size of all convolutional layers and transposed convolutional layers are 5*5 and 2*2, and the depths of the 8 convolutional layers are 64, 128, 256, 512, 512, 512, 512, 1024, the depths of the 8 transposed convolutional layers are 512, 512, 512, 512, 256, 128, 64, 1 in sequence.

In order to make the output of the generator more diverse and increase the robustness of the model, the random noise z is generated by the dropout operation in the generator, which is performed in the first three layers of the transposed convolution layer, and each node is reserved 50% chance.

Further, the discriminator is a two-dimensional neural network D, including 3 convolutional layers and 1 fully connected layer, the filter size of the 3 convolutional layers is 5*5, and the step size is 2*2 , the filter depths are 64, 128, and 256 respectively, the fully connected layer uses sigmoid as the activation function, and the output result is a one-dimensional scalar.

In order to make a good trade-off between recall rate and false alarm rate, the loss function of the discriminator is:

The loss function of the generator is:

in, L _s = ‖G(x,z)‖ ₁ , ‖·‖ ₁ is the expression of the L ₁ norm; D(x,y) is the output of the discriminator, G(x,z) is the output of the generator, is the expectation of the probability distribution of the true sample pair, is the expectation of the probability distribution of the false sample pair, λ and β are respectively and L _s influence the weight of the entire loss function.

Further, the manual labeling method in step (1) is to label the target area in each terahertz graph, and label the pixel value of the target area as 255 and the rest of the positions as 0. The preprocessing process is to label all pixels. The value is converted to between -1 and 1, and the conversion formula is: p/127.5–1, where p is the specific pixel value.

The computer storage medium of the present invention stores a computer program thereon, and when the computer program is executed by a computer processor, any one of the methods described above is implemented.

Beneficial effects: the present invention can train an accurate detection and segmentation model under the condition that the number of training samples is small and the signal-to-noise ratio of the picture is very low, and the model has fast processing speed and high timeliness, It can be used in real-time security inspection systems, thereby reducing the labor intensity of manual security inspections and improving efficiency.

Description of drawings

1 is a schematic diagram of a selected terahertz image;

Fig. 2 is the schematic diagram of this method model training and testing process;

Fig. 3 is the comparative schematic diagram of the model structure used and other schemes;

Fig. 4 is the detailed structure of the model of the present invention.

Detailed ways

The schematic diagram of the terahertz image is shown in Figure 1, where the target object is partially surrounded by a rectangular frame. The specific process of this method is shown in Figure 2, which is divided into a training process and a testing process. After training, the purpose is to establish a trained model. In the testing process, the images to be tested are input into the model to test the effect of the method. The generative confrontation network adopted in the present invention is a generative model, and the process of transforming from one picture to another can be realized through the confrontation process. A standard generative adversarial network consists of a generator and a discriminator. Among them, the task of the generator is to generate sufficiently realistic fake samples to deceive the discriminator based on the input information. On the other hand, the discriminator accepts both real samples sampled from the data set and fake samples generated by the generator and then determines which ones The samples are real and which are fake. Finally, the generator can generate samples that the discriminator cannot discriminate between true and false, that is, the generator captures the distribution of training samples. The condition-based generative adversarial network constrains the generator with additional information, and can generate good results according to the user's labeled data even under the condition of a small sample size, which is the design idea adopted by the present invention.

During the training process, the correct detection and segmentation results are provided for each sample. Each image corresponds to an image of the same size, with all pixel values 0. Pixels are marked with a value of 255; samples that do not contain any weapons are left unmodified. Specifically, the target area in each terahertz graph is marked by manual labeling. The labeling tool uses a free labeling tool Colabeler to mark the pixel value of the target area as 255 and the rest as 0 to obtain the real segmentation. mask. After scaling the original terahertz image and the annotated image to 256 pixels in length and width, image preprocessing begins. The preprocessing process is to set all pixel values between -1 and 1, and the conversion formula is: p/127.5–1 (p is a specific pixel value). The random noise z is input into the generator, and the scaled original image to be detected is also input into the generator as the condition x. The generator learns the mapping function from the input image to the detection and segmentation results, and outputs the corresponding detection and segmentation results G(x,z). The input of the discriminator consists of two parts, one is a real sample pair: the original terahertz image x and its real segmentation, the detection result is the real segmentation mask y, denoted as (x, y); the other is a fake sample Right: the original terahertz image x and the generator's output pseudo-segmentation mask G(x, z), denoted by (x, G(x, z)). For the discriminator, it needs to give higher scores to real pairs and lower scores to fake pairs, while for the generator, it needs to produce more and more real samples G(x,z ) to fool the discriminator so that (x,G(x,z)) gets a higher score, and in the end the discriminator cannot tell which of the two pairs of inputs is fake, so the generator learns an implicit Mapping of Hertz images to detection segmentation images. In the testing process, the image to be detected is input into the trained generator, then the model will discard the redundant information such as human body and background noise, and finally segment the hidden objects. Detection and segmentation are done simultaneously.

To achieve such a goal, the present invention first defines a suitable loss function.

The loss function of the discriminator is:

L _s = ‖G(x,z)‖ ₁ (3)

The loss function of the generator is:

Among them, ‖·‖1 is the expression of L ₁ norm _; D(x,y) is the output of the discriminator, the value is between 0 and 1, G(x,z) is the output of the generator, is the expectation of the probability distribution of the true sample pair, is the expectation of the probability distribution of the false sample pair, λ and β are respectively and L _s influence the weight of the entire loss function. Both x and y are sampled from the real sample distribution. The discriminator maximizes the expectation by making D(x,y) as close to 1 as possible and D(x,G(x,z)) as close to 0 as possible. That is, formula (1) is maximized. On the contrary, the generator tries to minimize the expectation by making D(x, G(x, z)) as close to 1 as possible. The noise z is represented as a dropout operation in the generator (randomly assigning the weights of some nodes to 0 during the training process), which makes the output of the generator more diverse and more fake samples, thus making the generator easier Obtain the original data distribution to increase model robustness. At the same time, x is added to the two networks as a condition, forming a conditional generative adversarial network, and the output of the constraint generator is paired with the input one by one, otherwise the results of detection and segmentation will be random.

Since formula (1) only considers the loss function of the generative adversarial network, its goal is to generate pictures that the discriminator cannot distinguish between true and false. However, being able to generate realistic images does not necessarily represent a good segmentation result. So formula (2) uses the Manhattan distance to measure the reconstruction error. The constraint of adding this item requires that the generated image not only be realistic enough, but also close enough to the correct segmentation result, that is, the accurate segmentation result. Also, using the Manhattan distance instead of the commonly used L ₂ norm is that the L ₂ constraint tends to lead to blurry generated images.

Like many object detection and segmentation tasks, the hidden weapons only occupy a small part of the whole image, so formula (3) requires that the generated images should also be sparse. The present invention uses the L ₁ norm as the sparsity constraint instead of the theoretical L ₀ norm and the commonly used L ₂ norm, because the former needs to solve NP-hard problems, while the latter can only approximate each item to a very small size is not guaranteed to be 0 for most elements.

The loss function of the generator is Equation (4). λ and β are respectively and L _s affect the weight of the entire loss function. In practice, λ and β are 800 and 5, respectively. The condition x and the reconstruction error can supervise the generation process to ensure that the generated image is in one-to-one correspondence with the input terahertz image, otherwise it is random and does not meet the tasks of target detection and segmentation. At the same time, accurate segmentation results are maintained by minimizing the Manhattan distance. As a priori knowledge, the sparsity constraint reduces the false alarm rate.

The training of the model uses gradient ascent to optimize the discriminator and gradient descent to optimize the generator. When training the generator, it is necessary to keep the parameters of the discriminator unchanged and minimize formula (4); while when training the discriminator, keep the current parameters of the generator unchanged and maximize formula (1). Train the discriminator and generator alternately in an iterative process until the model converges. In the specific implementation process, the learning rate is set to 0.00012, the minibatch is 8, and the entire training set is traversed as one iteration, and a total of 200 iterations are performed to obtain the final model.

The network structure has been improved. As shown in Figure 3, compared with the traditional "Encoder-Decoder" structure, the invention adds some connections between the decoding layer and the encoding layer. The specific generator structure is shown in Figure 4. , so as to obtain low-level features, which can improve the detection ability of the model for small targets. Compared to "U-Net", (Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer Assisted Intervention (2015) 234-241) new The model removes some connections that are close to the output layer. As shown in Figure 3, the output of the first few layers of the U-Net encoder contains a lot of edge and background information. If these layers are connected to the decoding layer close to the output layer, it will lead to higher false alarms, so this The invention removes this part of the connection and finds a balance between the recall rate and the false alarm rate. According to the above loss function and the corresponding model structure, the gradient descent algorithm is used to iteratively obtain the optimization result, and the final model parameters are obtained. The discriminator includes three convolutional layers and one fully connected layer. The filter sizes of the three convolutional layers are all 5*5, the strides are all 2*2, and the filter depths are 64, 128, and 256 respectively. . The fully connected layer uses sigmoid as the activation function to output the result as a one-dimensional scalar. As shown in Figure 4, the described generator contains 8 convolutional layers and 8 transposed convolutional layers. The filter size and moving step size of all convolutional layers and transposed convolutional layers are 5*5 and 2*2, and the depth of 8 convolutional layers and the depth of 8 transposed convolutional layers are: 64 , 128, 256, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 256, 128, 64, 1. The noise z is generated by dropout operation, which is performed in the first three layers of the transposed convolutional layers, respectively, and each node has a 50% chance of being preserved.

Using the model trained in the above steps, input a new image to be detected into the generator to obtain the corresponding detection and segmentation results. In the actual test, the model of the present invention can control the false alarm rate to 11.35% and at the same time increase the recall rate to 88.46%, which is obviously better than the existing detection and segmentation algorithms.

Embodiments of the present invention also provide a computer storage medium on which a computer program is stored. The aforementioned method of control can be implemented when the computer program is executed by a processor. For example, the computer storage medium is a computer-readable storage medium.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flows of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

Claims (7)

1. A method for detecting and segmenting a target object in a terahertz image is characterized by comprising the following steps:

(1) taking an original terahertz image x in a training set as a training sample, marking a target object region in x in a manual mode to obtain a real segmentation mask y, preprocessing x and y at the same time, and converting all pixel values in the image between-1 and 1;

(2) inputting the original terahertz image x and the random noise z into a generator, and generating a false segmentation mask by the generator;

(3) forming a real sample pair by the real segmentation mask and the corresponding original terahertz image, forming a false sample pair by the false segmentation mask and the corresponding original terahertz image, and inputting the false segmentation mask and the corresponding original terahertz image into a discriminator together for evaluation;

(4) respectively calculating loss functions of a generator and a discriminator, optimizing the generator by using a gradient descent algorithm to make the loss function of the generator as small as possible, optimizing the discriminator by using a gradient ascent algorithm to make the loss function of the discriminator as large as possible until the model converges, returning to the step (1), and entering the step (5) after traversing the training set;

(5) returning to the step (1) until the set iteration number is reached;

(6) and storing the model trained in the step, and inputting the new picture to be detected into a generator to obtain a detection and segmentation result corresponding to the new picture to be detected.

2. The method for detecting and segmenting the target object in the terahertz image according to claim 1, wherein: the generator is a two-dimensional neural network G and comprises 8 convolutional layers and 8 transposed convolutional layers, wherein the filter size and the moving step size of all the convolutional layers and the transposed convolutional layers are 5 x 5 and 2 x 2, the depths of the 8 convolutional layers are 64, 128, 256, 512 and 1024 in sequence, and the depths of the 8 transposed convolutional layers are 512,256,128,64 and 1 in sequence.

3. The method for detecting and segmenting the target object in the terahertz image according to claim 2, wherein: the random noise z is generated by the dropout operation in the generator, and is respectively performed in the first three layers of the transposed convolutional layer, and the probability of each node being reserved is 50%.

4. The method for detecting and segmenting the target object in the terahertz image according to claim 1, wherein: the discriminator is a two-dimensional neural network D and comprises 3 convolution layers and 1 full-connection layer, the size of a filter of each convolution layer is 5 x 5, the step length is 2 x 2, the filter depth is 64, 128 and 256, the full-connection layer uses sigmoid as an activation function, and the output result is a one-dimensional scalar.

5. The method of detecting and segmenting a target object in a terahertz image according to claim 1, wherein the loss function of the discriminator is:

the loss function of the generator is:

wherein ,L_s＝||G(x，z)||₁，||·||₁is L₁A norm expression; d (x, y) is the output of the discriminator, G (x, z) is the output of the generator,as an expectation of the probability distribution of the true sample pairs,for the expectation of the probability distribution of the false sample pairs, λ and β are and L_SThe weight of the whole loss function.

6. The method for detecting and segmenting the target object in the terahertz image according to claim 1, wherein: the manual labeling method in the step (1) is to label a target region in each terahertz graph, label the pixel value of the target region as 255, and set the rest positions as 0, wherein the preprocessing process is to convert all the pixel values to between-1 and 1, and the conversion formula is as follows: p/127.5-1, where p is the specific pixel value.

7. A computer storage medium having a computer program stored thereon, characterized in that: the computer program, when executed by a computer processor, implements the method of any one of claims 1 to 6.