CN110135248A - A deep learning-based text detection method in natural scenes - Google Patents

Abstract

The invention discloses a natural scene text detection method based on deep learning. The method uses a CNN network to extract multi-scale features of text, and then uses RNN to encode these features to make full use of the contextual properties of text; then, the feature map is fed into the ROI pooling layer and outputs a series of text proposals. After non-maximum suppression, the generated text proposals are finally connected through a text connector, so as to realize multi-scale and multi-directional text detection flexibly and efficiently. The invention improves the accuracy and recall rate of text detection in natural scenes under the condition of multi-direction and variable scale.

Description

A deep learning-based text detection method in natural scenes

technical field

The invention belongs to the technical field of image processing, and in particular relates to a natural scene text detection method based on deep learning.

Background technique

Scene text detection is an important prerequisite for text recognition, and is often used in image retrieval, machine translation, autonomous driving and other fields. However, there are still many difficulties in the detection of text detection in complex background, multi-scale, multi-language, uneven illumination, blurred and so on.

Diversity and variability of natural scene text: Compared with the text in the document, the text in the natural scene may be multi-scale and multi-language, and the shape, direction, scale, and color may be different. brought many challenges.

Complex background: The scene text may appear in any background, including signal signs, bricks, grass, and fences. These backgrounds may have very similar characteristics to the text, which may become noise and affect the judgment of the text. At the same time, there is also the lack of text caused by the occlusion of foreign objects, resulting in potential detection errors.

Uneven imaging quality: Due to uncontrolled collection means, imaging quality cannot be guaranteed. The images used for detection may be distorted or out of focus due to different shooting angles or shooting distances, or noise and shadows may be formed due to different lighting during shooting.

For the problem of text detection in natural scenes, detection methods can be divided into two categories, one is the traditional detection method, and the other is the detection method based on deep learning. Traditional methods include texture-based methods, such as using local intensity, filter response, wavelet coefficients, etc.; there are region-based methods, such as stroke width transform (Stroke Width Transform, SWT), Maximum Stable Extremal Regions (Maximally Stable Extremal Regions). , MSER), stroke feature transform (StrokeFeature Transform, SFT) and so on. In recent years, with the development of deep neural networks, deep learning has shown more and more advantages in the field of computer vision. At present, the most popular deep learning method is based on Convolutional Neural Networks (CNN). After using deep learning, the accuracy of text detection is greatly improved, and people are freed from complex feature design work. Commonly used deep learning-based natural scene text detection models are usually based on common object detection models, such as RCNN, YOLO, SSD, etc. The basic structure of these models is usually to extract features with several convolutional layers and pooling layers, and finally use a fully connected layer for detection box classification and regression.

SUMMARY OF THE INVENTION

In order to more accurately and efficiently perform text detection in natural scenes and solve the problem of multi-directional and variable-scale text detection in natural scenes, the present invention proposes a natural scene text detection method based on deep learning.

The object of the present invention is achieved by at least one of the following technical solutions.

A natural scene text detection method based on deep learning, comprising the following steps:

(1) Build and train a neural network-based natural scene text detection model, including the following sub-steps:

(1.1) Construct a feature extractor based on Feature Pyramid Networks (FPN);

(1.2) Use Recurrent Neural Network (RNN) to encode the features extracted by the feature extractor;

(1.3) Use the ROI pooling layer to further improve the detection accuracy;

(1.4) Finally, the fully connected layer is used to classify and regress the detection frame to form a text detection model;

(1.5) Input the labeled training graph into the model; use a multi-task loss function including classification loss and regression loss to calculate the loss value to train the model;

(2) Use the natural scene text detection model completed by the above training to detect the natural scene text in a given image, including the following sub-steps:

(2.1) Input the image to be detected, use the above trained model to perform text version detection on the given image, and output a series of scores and coordinates of the text proposal detection box.

(2.2) Non-maximum suppression is performed on the obtained text proposals to remove partially redundant detection boxes.

(2.3) Use the text connector to connect a series of text proposals to generate the final detection result.

Compared with the prior art, the present invention has the following advantages and technical effects:

(1) The present invention uses Feature Pyramid Networks (FPN) for variable-scale text detection, which can efficiently utilize the information of convolutional layers of different sizes at the same time, compared to the method using the feature map of the last layer. , while using high-level strong semantic information and low-level high-resolution information to achieve higher recall rate and accuracy; compared with the method based on image pyramid, it greatly reduces the amount of computation.

(2) For multi-directional text detection, a series of text proposals are output, and finally these text proposals are connected through text connectors. Compared with the method of using any quadrilateral or rotating rectangle, fewer parameters are used. , so that the detection of multi-directional text is more flexible and efficient.

Description of drawings

FIG. 1 is a flowchart of text detection in a natural scene in an embodiment.

FIG. 2 is an architecture diagram of a natural scene text detection model used in the embodiment.

FIG. 3 is a diagram of actual results of detection in different scenarios using the text detection method of the present invention in the embodiment.

Detailed ways

In order to make the technical solutions and advantages of the present invention clearer, further detailed descriptions are given below with reference to the accompanying drawings, but the implementation and protection of the present invention are not limited thereto.

First, the terms in the present invention are explained:

Feature Pyramid Networks (FPN): FPN is directly modified on the original skeleton network, and the feature map of each resolution is introduced into the feature map of the next resolution scaled twice to do the corresponding addition operation of each element. Through such a connection, the feature maps used for each layer of prediction integrate features of different resolutions and different semantic strengths, and the fused feature maps of different resolutions are used for object detection of corresponding resolutions. This ensures that each layer has appropriate resolution and strong semantic features.

Residual network (ResNet): It is a deep convolutional network model proposed by He Kaiming in 2015. According to the number of layers used in the model, it is named ResNet-34, ResNet-50, ResNet-101, ResNet-152, etc.

Non-Maximum Suppression (NMS): Suppressing elements that are not maximum values can be understood as local maximum search. Each outputted detection box has a score, and these detection boxes may contain and intersect. NMS is used to select the detection boxes with the highest scores in the field, and suppress the detection boxes with low scores.

As shown in Figure 1, the natural scene text detection model based on deep learning in the present invention includes the following steps:

(1) Build and train a neural network-based natural scene text detection model, as shown in Figure 2, including the following sub-steps:

(1.1) Construct a feature extractor based on Feature Pyramid Networks (FPN). Using ResNet-101 as the skeleton network, a feature pyramid is generated, and the features of the layers from P2 to P5 are used.

(1.2) Use Recurrent Neural Network (RNN) to encode the extracted features. The extracted features are encoded using a Bi-directional Long Short-Term Memory (Bi-LSTM) recurrent neural network with 512 hidden layers as RNN.

(1.3) Using the ROI pooling layer to further improve the detection accuracy. The specific operation of ROI pooling is as follows:

(1.3.1) According to the input image, map the ROI to the corresponding position of the feature map;

(1.3.2) Divide the mapped area into parts of the same size, and the number of divisions is the same as the dimension of the output;

(1.3.3) Perform a max-pooling operation on each part.

(1.4) Finally, the fully connected layer is used for the classification and regression of the detection frame. The ROI pooled features are classified and regressed through two fully connected layers, respectively. If the number of output detection frames is k, the output dimension of the classification layer is 2k, corresponding to the text and background; the output dimension of the regression layer is 4k, corresponding to the upper left and lower right coordinates of the detection frame.

(1.5) Input the labeled training graph to train the model. Among them, the training images can be marked with quadrilaterals or rectangles. However, before inputting the model, it needs to be divided according to the given width. If the training image is marked as a quadrilateral, the smallest circumscribed rectangle after the segmentation is taken; if it is marked as a rectangle, it is directly divided.

Design a multi-task loss function that includes classification loss and regression loss. Use the designed loss function to calculate the loss:

where L, L _cls , and L _reg are the total loss, classification loss, and regression loss, respectively, and λ is the weight coefficient that balances the classification loss and regression loss. p _i is the category predicted by the ith detection box, is the ground-truth category of the ith detection box. t _i is the predicted coordinate of the ith detection box, are the true coordinates of the ith detection box.

(2) Use the natural scene text detection model completed by the above training to detect the natural scene text in a given image, including the following sub-steps:

(2.1) Input the image to be detected, use the above-trained model to perform text detection on the given image, and output a series of scores and coordinates of text proposal detection boxes.

(2.2) Non-maximum suppression is performed on the obtained text proposals to remove partially redundant detection boxes. The specific operations are as follows:

For the list B of text proposal detection boxes and their corresponding scores S, the following calculation methods are used. Choose the one with the largest score

The detection frame M is removed from the B set and added to the final detection result D. Usually the remaining detection box in B

Boxes with an IoU greater than a threshold with M are removed from B. Repeat this process until B is empty.

(2.3) Use the text connector to connect a series of text proposals to generate the final detection result. use as

The following steps perform concatenation of text proposals:

If the proposal P _j and the proposal P _i (where i and j represent different proposals) satisfy the following two conditions, the proposal P _j is defined as the neighbor of the proposal P _i :

(1) The proposal P _j and the proposal P _i are the closest and the distance between them is less than w _j + w _i

(2) The proposal P _j and the proposal P _i have a degree of coincidence greater than 0.5 in the vertical direction

where _wi and w _j are the widths of proposal Pi and proposal P _j respectively, if proposal _Pi is a neighbor of proposal P _j and proposal P _j is a neighbor of proposal _{Pi ,} this connects these two proposals into the same detection frame. Repeat the above steps until all proposed connections are completed, and the detection frame is the final output result. It can be seen from FIG. 2 and FIG. 3 that the detection effect of the present invention in a natural scene shows that the present invention can perform good detection on text with variable scales and multiple directions in a natural scene.

Claims (8)

1. a kind of natural scene Method for text detection based on deep learning, it is characterised in that the following steps are included:

(1) it constructs and trains natural scene text detection model neural network based, comprising:

(1.1) building is based on the feature extractor of feature pyramid network (Feature Pyramid Networks, FPN)；

(1.2) spy that feature extractor is extracted using Recognition with Recurrent Neural Network (Recurrent Neural Network, RNN) Sign is encoded；

(1.3) precision of detection is further increased using the pond ROI layer；

(1.4) classification and recurrence that detection block is finally carried out using full articulamentum, form text detection model；

It (1.5) will be by the training figure input model of mark；

(1.6) penalty values are calculated with training pattern using the multitask loss function comprising Classification Loss and recurrence loss；

(2) the natural scene text in given image is examined using the natural scene text detection model that training is completed It surveys, includes following sub-step:

(2.1) image to be detected is inputted, text inspection is carried out to given image using the natural scene text detection model after training It surveys, exports score and coordinate that a series of texts propose detection block；

(2.2) obtained text is proposed to carry out non-maxima suppression, to remove partial redundance detection block；

(2.3) proposal of a series of text is attached using text connector, generates final testing result.

2. the natural scene Method for text detection according to claim 1 based on deep learning, which is characterized in that constructing In natural scene text detection model neural network based, feature pyramid network (Feature Pyramid Networks, FPN the level from P2 to P5) has been only used.

3. the natural scene Method for text detection according to claim 1 based on deep learning, which is characterized in that constructing In natural scene text detection model neural network based, feature pyramid network (Feature Pyramid Networks, FPN) used ResNet-101 as back bone network.

4. the natural scene Method for text detection according to claim 1 based on deep learning, which is characterized in that constructing In natural scene text detection model neural network based, nerve is recycled using the two-way long short-term memory of 512 hidden layers Network (Bi-directional Long Short-Term Memory, Bi-LSTM) is used as Recognition with Recurrent Neural Network (Recurrent Neural Network, RNN) feature extracted is encoded.

5. the natural scene Method for text detection according to claim 1 based on deep learning, which is characterized in that constructing In natural scene text detection model neural network based, the calculating lost using following loss function:

Wherein L, L_clsAnd L_regRespectively total losses, Classification Loss and recurrence loss, λ are balanced sort loss and recurrence loss Between weight coefficient,It is the true classification of i-th of detection block.

6. the natural scene Method for text detection according to claim 5 based on deep learning, which is characterized in that classification damage Mistake is defined as follows:

Wherein, p_iIt is the prediction classification of i-th of detection block,It is the true classification of i-th of detection block.

7. the natural scene Method for text detection according to claim 5 based on deep learning, which is characterized in that return damage Mistake is defined as follows:

Wherein, t_iIt is the prediction coordinate of i-th of detection block,It is the true coordinate of i-th of detection block.

8. the natural scene Method for text detection according to claim 1 based on deep learning, which is characterized in that giving Determine to use following steps to carry out the connection of text proposal during the natural scene text in image detected:

If proposing P_jWith proposal P_iMeet following two conditions, will propose P_jIt is defined as proposing P_iNeighbours:

(1) propose P_jWith proposal P_iIt is nearest and they the distance between be less than w_j+w_i；

(2) propose P_jWith proposal P_iThere is the registration greater than 0.5 in vertical direction

Wherein w_iAnd w_jRespectively propose P_iWith proposal P_jWidth, if propose P_iIt is to propose P_jNeighbours and propose P_jIt is to mention Discuss P_iNeighbours, the two proposals are connected as the same detection block by this.