CN110334703A - A Ship Detection and Recognition Method in Day and Night Images - Google Patents

Abstract

The present invention proposes a ship detection and recognition method in day and night images, including the following steps: S1, using light sensing elements to detect the illuminance of ship images in different periods, and dividing them into daytime images and nighttime images according to different illuminance ranges of ship images S2. For daytime images, first detect all objects that appear within the detection range, and then filter out ship-like objects; S3. For night-time images, first detect prominent targets in night-time images, and then filter out ship-like objects ; S4. Obtain the real-time position and category information of all ships in the current video frame based on the filtered ship objects. The method of the invention realizes the detection and recognition of the ship target in the scene of the whole time, and has better robustness.

Description

A Ship Detection and Recognition Method in Day and Night Images

technical field

The invention relates to the fields of computer vision and digital image processing, in particular to a ship detection and recognition method in day and night images based on statistical learning and regional covariance.

Background technique

Target detection is to find all the objects of interest in the image, including two sub-tasks of object positioning and object classification, that is, to determine the category and position of the object at the same time. Object detection is a popular direction in computer vision and image processing, and it is widely used in many fields such as robot navigation, intelligent video surveillance, and industrial inspection. It is of great practical significance to reduce the consumption of human capital through computer vision. Therefore, target detection has become a research hotspot in theory and application in recent years. It is an important branch of image processing and computer vision, and it is also the core part of intelligent monitoring system. At the same time, target detection is also a basic algorithm in the field of pan-identification, which plays a vital role in subsequent tasks such as face recognition, gait recognition, crowd counting, and instance segmentation.

Due to the widespread use of deep learning, target detection algorithms have developed relatively rapidly. Since 2006, under the leadership of Hinton, Bengio, Lecun, etc., a large number of papers on deep neural networks have been published. Especially in 2012, Hinton's research group participated in the ImageNet image recognition competition for the first time, and won the prize in one fell swoop through the constructed CNN network AlexNet. won the championship, and since then the neural network has received widespread attention. Deep learning uses multi-layer computing models to learn abstract data representation, and can discover complex structures in big data. At present, this technology has been successfully applied to various pattern classification problems including computer vision.

The analysis of computer vision for target motion can be roughly divided into three levels: motion segmentation, target detection; target tracking; action recognition, behavior description. Among them, target detection is not only one of the basic tasks to be solved in the field of computer vision, but also a basic task of video surveillance technology. Since the targets in the video have different postures and often occlude, and their motion is irregular, at the same time, considering the depth of field, resolution, weather, illumination and other conditions of the surveillance video and the diversity of the scene, the results of the target detection algorithm will be directly Affect the effect of subsequent tracking, action recognition and behavior description. Therefore, even with the development of technology today, the basic task of object detection is still a very challenging topic, and there is great potential and room for improvement.

At present, the target detection and recognition method based on deep learning is used in the detection and recognition of ships. The daytime scene performs well, but for the night scene, due to the large difference in the illumination, contrast and signal-to-noise ratio of the night image, the night ship The performance of detection and recognition drops sharply. In order to intelligently detect the position of the ship from the full-time video surveillance and automatically identify the type of the target ship, the key point is to extract the image features of the ship. However, in practical applications, the signal-to-noise ratio and contrast of images at different times of the day and night are quite different, which poses a great challenge to the image feature extraction of ships.

The current mainstream target detection algorithms based on deep learning models can be divided into two categories: One-Stage and Two-Stage. Usually, the One-Stage detection algorithm, which does not require the Region Proposal stage, directly generates the category probability and position coordinate value of the object, and the speed is relatively fast; the Two-Stage target detection algorithm divides the detection problem into two stages, first generating Candidate regions (region proposals), and then classify and refine the location of the candidate regions. Compared with the previous one, this type of algorithm has greatly improved the accuracy, but the speed is relatively slow.

Contents of the invention

In order to realize the detection and recognition of the water surface target ship at all times, the present invention proposes a ship detection and recognition method in day and night images, which includes the following steps:

S1. Use the photosensitive element to detect the illuminance of ship images in different periods, and divide them into daytime images and nighttime images according to the different illuminance ranges of ship images;

S2. For daytime images, firstly detect all objects appearing in the detection range, and then filter out ship-like objects;

S3. For the nighttime image, firstly detect the significant target in the nighttime image, and filter out the ship-like object therefrom;

S4. Obtain real-time positions and category information of all ships in the current video frame based on the filtered ship objects.

Further, step S1 specifically includes:

S11. Collect a large number of scene pictures in different time periods, statistically analyze the image illuminance ranges in each time period, and form a reference comparison table for illuminance ranges;

S12. Detect the illuminance of the ship image from the camera through the photosensitive element, compare the illuminance range and refer to the comparison table to determine whether the type of the ship image is a daytime image or a nighttime image.

Further, in step S2, use the deep convolutional neural network-based target detection algorithm Faster R-CNN to process the daytime image, and its network structure includes two parts: RPN and Fast R-CNN, where RPN is used to predict that the input image may contain The candidate area of the target, output the proposal box that may contain the ship target; Fast R-CNN is used to classify the candidate area, and correct the bounding box of the candidate area.

Further, the training steps of the target detection algorithm Faster R-CNN based on the deep convolutional neural network are as follows:

1) Initialize the RPN network parameters with the pre-trained network model, and fine-tune the RPN network parameters through the stochastic gradient descent algorithm and the back propagation algorithm;

2) Initialize the Faster R-CNN target detection network parameters with the pre-trained network model, and use the RPN network in the first step to extract candidate regions and train the target detection network;

3) Re-initialize and fine-tune the RPN network parameters with the target detection network in the second step;

4) Use the RPN network in the third step to extract candidate regions and fine-tune the target detection network parameters;

5) Repeat steps 3 and 4 until the maximum number of iterations is reached or the network converges.

Further, step S2 specifically includes:

S21. Calculate the convolution feature map of the daytime image to be detected;

S22. Using RPN to process the convolutional feature map to obtain a target suggestion frame;

S23. Using RoI Pooling to extract features for each suggestion frame;

S24. Classify by using the extracted features.

Further, in step S3, the nighttime image is processed using a convolutional neural network algorithm guided by region covariance.

Further, step S3 specifically includes:

S31. Extracting low-level features of nighttime images in units of pixels;

S32. Constructing regional covariance based on multidimensional feature vectors;

S33. Using the covariance matrix as a training sample to construct a convolutional neural network model;

S34. Calculate image saliency based on local and global contrast principles;

S35 , frame a prominent ship target, and acquire the position of the ship.

Further, the ship detection and identification method of the present invention also includes:

S5. Use the AUC and MAE evaluation indicators to judge the image detection results; the calculation formulas of AUC and MAE are as follows:

Among them, rank _insi represents the serial number of the i-th sample, which indicates that the probability score is ranked from small to large, and ranks at the rank position. M and N are the number of positive samples and the number of negative samples, respectively. Indicates that only the serial numbers of the positive samples are added;

in represents a significant map, Represents the benchmark map, W and H represent the pixel value width and height of the image, respectively.

The beneficial effects of the present invention are as follows:

The ship detection and recognition method of the present invention classifies the images based on different illuminances of the ship images, and uses different processing strategies for the classified daytime images and nighttime images, so that the method of the present invention can be used even when the image quality is poor Most of the ships can also be detected on nighttime images, and even if the scale of the ship changes, it can be detected, thus realizing the detection and recognition of ship targets in the full-time scene, which has good robustness.

Description of drawings

Fig. 1 is a basic flowchart of an embodiment of the ship detection and identification method of the present invention.

Fig. 2 is an example of day and night ship images in the embodiment of the present invention.

FIG. 3 is a flow chart of the Faster R-CNN target detection algorithm used in the embodiment of the present invention.

Fig. 4 is an implementation effect diagram obtained by using a ship model to simulate the movement test of a real ship in an embodiment of the ship detection and recognition method of the present invention, wherein: a is an image of a distant ship, b is an image of a nearby ship, and c is an image of multiple ships Obstacle ship image, d is the ship scale transformation image.

Fig. 5 is a block diagram of a convolutional neural network guided by region covariance used in an embodiment of the present invention.

Fig. 6 is an effect diagram of the ship detection and recognition method of the present invention obtained by using a ship model to simulate the movement of a real ship in a lake on campus at night.

Detailed ways

In order to further understand the present invention, the preferred embodiments of the present invention are described below in conjunction with examples, but it should be understood that these descriptions are only to further illustrate the features and advantages of the present invention, rather than limiting the claims of the present invention.

An embodiment of the present invention provides a ship detection and recognition method in day and night images based on statistical learning and regional covariance. As shown in Figure 1, its process includes:

1. Firstly, the video frame image obtained through the photoelectric head is detected by the photosensitive element to realize the classification of day and night images;

2. For daytime and nighttime images, use Faster RCNN and region-guided covariance to guide CNN to detect the size and position of the ship. The specific processes are shown in Figure 3 and Figure 5 respectively;

3. Screen the detected ships to determine the type and location of the ships.

In step 1, examples of day and night ship images are shown in Figure 2, where the first one is a daytime image and the second one is a nighttime image. The specific process of using light-sensing elements to realize day-night image classification is as follows: First, collect a large number of scene pictures in different time periods, and statistically analyze the image illuminance range of each time period. The detailed parameters are shown in Table 1. The left side is the daytime scene illuminance range, and the right side is the nighttime scene illuminance range. scope. Through the light-sensing element detection method, that is, the ordinary light-sensing element detects the illuminance of the image sent by the camera, and compares the range of reference values in the table below to determine the category of the image, that is, a daytime image or a nighttime image.

Table 1. Reference values of illuminance ranges in various natural periods

Natural conditions Illuminance value (Lx) Natural conditions Illuminance value (Lx) direct sunlight (1～1.3)×10⁵ deep dusk 1 full daylight (1～2)×10⁴ full moon 10^-1 daytime (overcast) 10³ crescent moon 10^-2 very dark day 10² starlight 10^-3 dusk (dawn) 10 night (overcast) 10^-4

In step 2, the main steps of the Faster R-CNN (Region-based Convolutional Neural Networks, faster region-based convolutional neural network) target detection algorithm for ship detection are:

1) Calculate the convolution feature map of the ship image;

2) Use RPN (Region Proposal Network, Region Proposal Network) to process the convolutional feature map to obtain the target proposal frame;

3) Use RoI Pooling (Region of interest pooling, region of interest pooling) to extract features for each suggestion frame;

4) Use the extracted features to classify.

For the ship target detection and recognition in the daytime scene, the data set selected in this embodiment is the ship picture data set and ship model data set in the Yangtze River actually taken by the Yingwuzhou Bridge. The training data set and the other part as the test set. Among them, ships are divided into five categories, namely passenger ships, cargo ships, light ships, warships and sailing ships. The pre-training model selected in this embodiment is ResNet50. RPN is trained end-to-end during the training phase. The initial learning rate in the Faster R-CNN network is 0.0003, and iterates 20,000 times. The specific training steps are as follows:

1) Initialize the RPN network parameters with the pre-trained network model, and fine-tune the RPN network parameters through the stochastic gradient descent algorithm and the back propagation algorithm;

2) Initialize the Faster R-CNN target detection network parameters with the pre-trained network model, and use the RPN network in the first step to extract candidate regions and train the target detection network;

3) Re-initialize and fine-tune the RPN network parameters with the target detection network in the second step;

4) Use the RPN network in the third step to extract candidate regions and fine-tune the target detection network parameters;

5) Repeat steps 3 and 4 until the maximum number of iterations is reached or the network converges.

The performance of the model is verified on the test set, and the statistics of the false negative rate and false negative rate indicators are obtained, as shown in Table 2.

Table 2 Faster R-CNN model false negative rate

Indicator type cargo ship passenger ship light ship warship sailboat False negative rate 0.221 0.117 0.667 0.212 0.006 False alarm rate 0.051 0.072 0.015 0.103 0.077

In the experimental effect diagram shown in Fig. 4, for the multi-ship regression diagram on the lower left, Fig. 4(c), the following explanations are made: the resolution of the image is 233×151, and the white foam is the obstacle on the water surface, and the parameter values of the algorithm operation results Table 3 below. The other three pictures are similar to those in Figure 4(c).

Table 3 Ship position type information table

vessel number Coordinate information (x,y) width&eight type Confidence Ship 1 (Left 1) (25,101) 21&12 Speed-boat 84% Ship 2 (Left 2) (47,82) 21&13 Speed-boat 76% Ship 3 (Left 3) (121,29) 28&26 Pump-ship 99% Ship 4 (Left 4) (210,77) 12&12 Speed-boat 89%

In step 2, for nighttime video frame images, in order to solve the problem of unbalanced training samples caused by single visual information, the embodiment of the present invention proposes a convolutional neural network algorithm based on region covariance guidance, which is used to detect notable target. Since the salient target detection is a research proposed by simulating the human visual attention mechanism, the area of most interest to the human eye is used as the detection object. In the scene of a ship sailing on the water with a relatively simple background, the ship object is a salient target. The ship position is obtained by returning the bounding box of the salient ship object. As shown in Figure 5, the main steps of the convolutional neural network algorithm based on region covariance guidance are:

1) Extract the low-level features of the image in units of pixels;

2) Construct the regional covariance based on the multidimensional feature vector;

3) Construct a convolutional neural network model with the covariance matrix as a training sample;

4) Calculate image saliency based on local and global contrast principles;

5) Frame a significant ship target to obtain the ship's position.

The model training of the nighttime scene is different from that of the daytime scene, but its training and testing steps can refer to the training scheme of the Faster R-CNN target detection algorithm in step 2 and the training and testing steps in Figure 3. The data set used in this module is the same night time period imagery as the daytime scene location. Due to the particularity of the night scene and the characteristics of the algorithm used in this module, the evaluation criteria selected by this module are the mainstream AUC and MAE evaluation indicators in this field, and the running time unit for each image is: seconds. The specific index values are shown in Table 4.

The calculation formulas of AUC and MAE are as follows:

Among them, rank _insi represents the serial number of the i-th sample (the probability score is ranked from small to large, ranking in the rank position), M and N are the number of positive samples and the number of negative samples, respectively, Indicates that only the serial numbers of the positive samples are added up.

in represents a significant map, Represents the benchmark map, W and H represent the pixel value width and height of the image, respectively.

Table 4 Evaluation index of ship detection algorithm at night

Indicator name MAE AUC Time Index value 0.1329 0.8546 1.553

It can be seen from Table 4 that during the night time period, due to the lack of ship image information, the real-time effect cannot be achieved when the ship image is saliently processed, but the ship target detection function can be better realized. MAE is the mean absolute error, and the smaller the value, the better the performance of the algorithm. AUC is a probability value, which can intuitively evaluate the quality of the classifier, and the larger the value, the better.

The realization effect obtained by using the ship model to simulate the real ship motion test in the lake on campus at night time is shown in Figure 6, and the multi-ship regression graph on the lower right is explained as follows: the resolution of the picture is 233×155, and the ship information is the result of the algorithm operation The output parameter values are shown in Table 5. The other three pictures are similar.

Table 5 Ship position type information table

Ship model number Coordinate information (x,y) width height Confidence Ship 1 (Left 1) (75,97) 52 50 94% Ship 2 (Left 2) (128,41) 37 25 86%

It can be seen from the detection results that the ship detection model of the embodiment of the present invention can detect most ships even on nighttime images with poor image quality, and can be detected even if the scale of the ship changes. To sum up, the present invention realizes the detection and recognition of ship targets in a full-time scene, and has better robustness.

The descriptions of the above embodiments are only used to help understand the method and core idea of the present invention. It should be pointed out that for those skilled in the art, without departing from the principle of the present invention, some improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims (8)

1. ship detecting and recognition methods in a kind of image round the clock, which comprises the steps of:

S1, using light sensitive component detection different periods ship image illumination, according to the different illumination ranges of ship image by its It is divided into two class of day images and nighttime image；

S2, day images are directed to, all objects appeared in investigative range is detected first, are then screened out from it ship Oceangoing ship type objects；

S3, it is directed to nighttime image, the first well-marked target in detection nighttime image, is screened out from it ship type objects；

S4, based on the ship type objects filtered out, obtain the real time position and generic of all ships in current video frame Information.

2. ship detecting and recognition methods in image round the clock according to claim 1, which is characterized in that step S1 is specific Include:

S11, a large amount of different periods scene pictures of acquisition, statistical are precipitated the image illumination range of day part, form illumination range With reference to the table of comparisons；

S12, the illumination that the ship image that camera transmits is detected by light sensitive component compare the illumination range with reference to the table of comparisons The classification for judging ship image is day images or nighttime image.

3. ship detecting and recognition methods in image round the clock according to claim 1, which is characterized in that in step S2, Day images are handled using the algorithm of target detection Faster R-CNN based on depth convolutional neural networks, network structure includes Two parts RPN and Fast R-CNN, wherein RPN is used to predict in input picture to export comprising the candidate region of target It may include the Suggestion box of ship target；Fast R-CNN corrects the boundary of candidate region for the candidate region of classifying Frame.

4. ship detecting and recognition methods in image round the clock according to claim 3, which is characterized in that it is described based on The training step of the algorithm of target detection Faster R-CNN of depth convolutional neural networks is as follows:

1) RPN network parameter is initialized with pre-training network model, it is micro- by stochastic gradient descent algorithm and back-propagation algorithm Adjust RPN network parameter；

2) Faster R-CNN target detection network parameter is initialized with pre-training network model, and with the RPN net in the first step Network extracts candidate region, and training objective detects network；

3) it is reinitialized with the target detection network in second step and finely tunes RPN network parameter；

4) candidate region is extracted with the RPN network in third step and target detection network parameter is finely adjusted；

5) third step and the 4th step are repeated, until reaching maximum number of iterations or network convergence.

5. ship detecting and recognition methods in image round the clock according to claim 3, which is characterized in that step S2 is specific Include:

S21, the convolution characteristic pattern for calculating day images to be detected；

S22, the convolution characteristic pattern is handled using RPN, obtains target Suggestion box；

S23, feature is extracted to each Suggestion box using RoIPooling；

S24, classified using the feature of extraction.

6. ship detecting and recognition methods in image round the clock according to claim 1, which is characterized in that in step S3, Use the convolutional neural networks algorithm process nighttime image guided based on region covariance.

7. ship detecting and recognition methods in image round the clock according to claim 6, which is characterized in that step S3 is specific Include:

S31, the low-level features that nighttime image is extracted using pixel as unit；

It S32, take multidimensional characteristic vectors as basis structure realm covariance；

S33, convolutional neural networks model is constructed by training sample of covariance matrix；

S34, saliency is calculated based on part and global contrast principle；

S35, significant ship target is outlined, obtains vessel position.

8. the ship detecting in image and the recognition methods round the clock according to claim 5 or 7, which is characterized in that further include:

S5, image detection result is judged using AUC and MAE evaluation index；AUC and MAE calculation formula difference is as follows:

Wherein rank_insiThe serial number of i-th sample is represented, indicates that probability score is arranged from small to large, comes the rank position, M, N is the number of positive sample and the number of negative sample respectively,It indicates only to add up the serial number of positive sample；

WhereinIndicate significant map,Indicate that benchmark map, the pixel value that W and H respectively indicate image are wide and high.