JP2021022367A - Image processing method and information processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing method sum information processing apparatus. An image processing method inputs a processing-waiting image containing text into a convolutional neural network model for character recognition, extracts text features; and uses the extracted text features as a convolutional neural network model. Convolutional neural network models and recursive neural network models include inputting into a recursive neural network model for connected word meaning recognition and recognizing text in awaiting images, and convolutional neural network models and recursive neural network models connect to each other with a definite waiting layer. Based on the initial convolutional neural network model that has and the initial recursive neural network model that has a definite waiting layer, each definite waiting layer is searched in the predefined candidate model space and connected to each other in an end-to-end manner. Obtained by performing joint training on a convolutional neural network model and a recursive neural network model. [Selection diagram] Fig. 1

Description

The present invention relates to the field of image processing, and more particularly to an image processing method for recognizing text contained in an image and an information processing apparatus capable of realizing the image processing method.

Recognition of text in images, such as handwritten character recognition (OCR), is one of the research topics in the field of computer vision.

So far, much of the English-based research is related to deep convolutional neural networks (DCNNs). In these studies, for example, recognition of text consisting of handwritten characters is regarded as one image classification problem, and one classification label (there are about 90,000 words in total) is assigned to each English word. This is a large training model with a large amount of categorization. Since the number of basic combinations of such word sequences exceeds one million, it cannot be extended to characters in other languages, such as Chinese text, Japanese text, and so on. Therefore, the DCNN-based English character recognition system cannot be directly used for image-based text sequence recognition. For example, if such an English character recognition system is to be diverted to the recognition of a Chinese character sequence, a large amount of manual (artificial) design is required for another design and training.

Therefore, a method capable of recognizing the text included in the image by using a model that is easier to acquire and suitable for character recognition of various languages (for example, Chinese character recognition) is desirable.

As described above, in view of the desire for a method of recognizing text contained in an image using a model that is easier to obtain, an object of the present invention is obtained by a search and end-to-end training method. Further, to provide an image processing method capable of performing text recognition using a convolutional neural network (CNN) and a recurrent neural network (RNN) connected to each other, and an information processing device capable of realizing the image processing method. There is.

According to one aspect of the present invention, an image processing method is provided, the image processing method.
Input awaiting image containing text into a convolutional neural network model for character recognition, extract text features; and recurrently connect the extracted text features with a convolutional neural network model. Includes inputting into a type neural network model and recognizing text in awaiting images
The convolutional neural network model and the recurrent neural network model are in a predefined candidate model space based on the initial convolutional neural network model having a definite waiting layer and the initial recurrent neural network model having a definite waiting layer connected to each other. It is obtained by searching for each fixed waiting layer in, and performing joint training on convolutional neural network models and recurrent neural network models connected to each other in an end-to-end manner.

According to another aspect of the present invention, an information processing device is provided, the information processing device includes a processor, and the processor includes a processor.
Input awaiting image containing text into a convolutional neural network model for character recognition, extract text features; and recurrently connect the extracted text features with a convolutional neural network model. Constructed to enter into a type neural network model and recognize text in awaiting images,
The convolutional neural network model and the recurrent neural network model are in a predefined candidate model space based on the initial convolutional neural network model having a definite waiting layer and the initial recurrent neural network model having a definite waiting layer connected to each other. It is obtained by searching for each fixed waiting layer in, and performing joint training on convolutional neural network models and recurrent neural network models connected to each other in an end-to-end manner.

According to another aspect of the present invention, there is provided a program that causes a computer to perform the image processing method as described above.

According to another aspect of the invention, a corresponding storage medium is further provided, in which a machine-readable command code is stored, which command code is read by a machine (eg, a computer). When executed, the machine can be made to perform the image processing method described above.

According to each aspect of the present invention described above, at least the following advantages can be obtained, that is, when recognizing text in a waiting image, it is acquired by a search and end-to-end training method. CNN and RNN models that are connected to each other are utilized, and such models can reduce manual intervention in the model building process of the previous period, thus reducing the cost of acquiring the model. be able to.

It is a flowchart of the exemplary flow of the image processing method in the Example of this invention. It is a figure which shows the general architecture for explaining the connected CNN model and RNN model used in the image processing method shown in FIG. It is a figure for demonstrating the concrete structure of the architecture shown in FIG. It is a figure which shows the confirmation waiting layer in the CNN model in FIG. It is a figure which shows the confirmation waiting layer in the RNN model in FIG. It is a flowchart of an exemplary process for acquiring a CNN model and an RNN model connected to each other used in the image processing method shown in FIG. It is a block diagram which shows the exemplary structure of the image processing apparatus which concerns on embodiment of this invention. It is a figure which shows the hardware structure which can realize the image processing method and apparatus according to the Example of this invention.

Hereinafter, suitable examples for carrying out the present invention will be described in detail with reference to the attached drawings. It should be noted that such an example is merely an example and does not limit the present invention.

According to one aspect of the present invention, an image processing method is provided. Hereinafter, an exemplary flow of the image processing method according to the embodiment of the present invention will be described with reference to FIG.

FIG. 1 is a flowchart of an exemplary flow of the image processing method according to the embodiment of the present invention. As shown in FIG. 1, the exemplary flow 100 of the image processing method may include the following steps.

Step S101: Input awaiting image containing text into a convolutional neural network (CNN) model for character recognition and extract text features;
Step S103: The extracted text features are input to a recurrent neural network (RNN) model for word meaning recognition connected to a convolutional neural network model, and the text in the waiting image is recognized.

Here, the convolutional neural network (CNN) model used in step S101 and the recurrent neural network (RNN) model used in step S103 are an initial convolutional neural network model having a confirmation waiting layer and a confirmation layer connected to each other. Based on an early recurrent neural network model with a waiting layer, a search is performed for each fixed waiting layer in a predefined candidate model space, and a convolutional neural network model and a recurrent neural network connected to each other in an end-to-end manner. It is acquired by performing joint training on the network model.

For example, a fully convolutional recurrent neural network with interconnected CNN and RNN models acquired by the search and joint training schemes described above, and an optional transfer layer (eg, connectionist time classification (CTC) layer). (CRNN) Models can be constructed. So far, CRNN-based models are attracting attention as a method for recognizing text or character sequences in images. The advantages of the CRNN model over the traditional DCNN model include using fewer parameters and occupying less storage space. However, since all parts of the CNN in the conventional CRNN model are designed manually, it usually requires a large amount of a priori knowledge and early preparation.

On the other hand, the image processing method in this embodiment is composed of a CNN model and an RNN model connected to each other acquired by performing a search and end-to-end training in a predefined candidate model space. A CRNN model is utilized, which can significantly reduce manual intervention, thus reducing the cost of developing a text recognition method based on the model.

In one preferred embodiment, search and joint training with the loss function representing the probability of accurately recognizing the text and the optimization objectives of minimizing the overall complexity of the convolutional neural network model and the recurrent neural network model. By performing the above, a convolutional neural network model and a recurrent neural network model after optimization can be obtained. The setting of optimization goals, search and joint training will be explained in detail based on specific examples after explaining the architecture of the CNN model and RNN model connected to each other.

The model adopted in the image processing method in this preferred embodiment by performing search and joint training with a multi-target optimization method that optimizes the probability of accurately recognizing the text and the overall complexity of the model. Can simultaneously consider the quasi-accuracy of recognition and the scale or complexity of the model. Compared to a general recognition model that considers only recognition semi-accuracy, the model architecture with optimized overall complexity used in this example is relatively small in complexity or scale, so mobile platforms and resources. Especially suitable for restricted environments.

Subsequently, based on FIGS. 2 to 5, the structures of the connected CNN and RNN models, and the candidate model space and search for obtaining the model architecture, which are applied to the exemplary method shown in FIG. The operation will be described more specifically.

First, refer to FIG. FIG. 2 is an explanatory diagram of the general architecture of the connected CNN model and RNN model used in the image processing method shown in FIG. As shown in FIG. 2, the CRNN overall model 20 used in the exemplary method shown in FIG. 1 may include a sequentially connected CNN model 21, RNN model 22, and preferred CTC layer 23. In the prior art, there is already a method of applying the entire CRNN model as shown in FIG. 2 to the recognition of text in an image. For example, the paper “An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition” published by authors such as Baoguang Shi at IEEE Transactions on Pattern Analysis & Machine Intelligence in November 2017. By referring to, the principle, structure and concrete details of such a CRNN model can be understood. Therefore, the principle and structure of the overall model will be briefly described below for the convenience of subsequent explanations.

In the exemplary CRNN overall model 20 shown in FIG. 2, the CNN model 21 may have the basic structure of a general convolutional neural network, and may include, for example, a plurality of convolutional layers, pooling layers, and the like. It does not have a fully connected layer. CNN model 21 is used for feature extraction, it generates a feature map based on an input image containing, for example, the text "SUBWAY", and a recurrent neural network based on the generated feature map. By acquiring the feature sequence suitable for the input of, the feature sequence can be input to the RNN model 22. Since the feature sequence extracted by the CNN model 21 can represent the text in the image waiting to be processed, the feature sequence may be referred to as a "text feature" here.

Based on the feature sequence input from the CNN model 21 (ie, the "text feature" described above), the RNN model 22 recognizes the text sequence and outputs the prediction result. The prediction result output by the RNN model 22 may be the label distribution of each feature vector in the feature sequence input to it, that is, the probability list of the real result. The output of the RNN model 22 may include label distributions that correspond to noise, that is, inaccurate spaces, duplicate characters, and so on. To make it easier to understand, the example shown in FIG. 2 shows a character sequence containing noise, that is, "SSUBBWWAYY", which corresponds to the sequence label distribution output by the RNN model 22. In order to remove such noise, a CTC layer 23 as a transfer layer is provided after the RNN model 22, and the prediction of the RNN model 22 is converted into a final label sequence by operations such as duplication removal and integration. As shown in FIG. 2, after the processing of the CTC layer 23, the recognition result of the text sequence from which noise has been removed, for example, “SUBWAY” (that is, the probability distribution corresponding to the label sequence) can be obtained.

After understanding the CRNN overall model 20 including the sequentially connected CNN model 21, RNN model 22, and CTC layer 23 shown in FIG. 2, the specifics of the overall model are subsequently based on FIGS. 3 to 5. The structure, the candidate model space for obtaining the specific structure, and the search operation will be described.

First, refer to FIG. FIG. 3 is an explanatory diagram of the specific structure of the CRNN overall model shown in FIG. As shown in FIG. 3, the overall CRNN model may include a sequentially connected CNN model 21, RNN model 22, and preferred CTC layer 23. In FIG. 3, the first to Nth layers of the CNN model 21 are represented by a solid line frame without shading, the first to L layers of the RNN model 22 are represented by a solid line frame with shading, and the CTC layer 23 is a dotted frame. It is represented by. Here, N and L are all predetermined natural numbers, and indicate the corresponding number of layers (number of layers) in the CNN model 21 and the RNN model 22, respectively.

In one example, an initial network architecture with N = 12 layers in the CNN model and L = 2 layers in the RNN model may be configured for the application of character sequence recognition in images. In addition, based on the contents of the present invention, those skilled in the art will perform N and L in advance by experiments based on various factors such as complexity, accuracy, processing load, and calculation speed of the system. A specific value of may be reasonably set, but a detailed description thereof will be omitted here. Also, depending on the needs of the application, set other necessary parameters of the overall network architecture, such as the number of channels in the first layer of the CNN model related to the specific application or dataset, the number of channels in the last layer, and so on. You may (the number of channels in the middle layer can be calculated automatically), which allows you to build an early CNN model 21 and an early RNN model 22. Of course, based on the contents of the present invention, those skilled in the art may set the necessary parameters of the entire network architecture in advance by experiment based on factors such as system requirements, but the details are described here. The description is omitted.

In this embodiment, the first to Nth layers in the CNN model 21 and the first to Lth layers in the RNN model 22 may all be confirmation waiting layers. In other words, for the CRNN-wide model shown in FIG. 3, for example, from the input layer of the entire model architecture (that is, the first layer of the CNN model 21) to the output layer of the entire model architecture (for example, CTC layer 23). End-to-end training can determine the operation or structure of each layer in the CNN model 21 and each layer in the RNN model 22. Alternatively, the operation or structure of some layers in the CNN model 21 and the RNN model 22 may be determined in advance, and the remaining layers may be used as a confirmation waiting layer whose operation or structure is determined by searching. Unlike the CNN model 21 or the RNN model 22, the structure of the CTC layer 23 may be manually (artificially) determined in advance by a method such as a priori knowledge or experiment.

In one preferred embodiment, the predefined candidate model space may include multiple candidate convolution and candidate pooling operations for one definite waiting layer of the convolutional neural network model. Further, the operation of one convolutional neural network model can be confirmed by performing search and joint training in a plurality of candidate convolution operations and candidate pooling operations.

As an example, for one confirmed waiting layer in the CNN model 21 shown in FIG. 3, the candidate convolution operation may include a perforated convolution operation. Dilated convolutions may also be referred to as dilated convolutions, which extend the size of the convolution kernel by filling with holes to obtain a larger field of view or receptive field, thereby image. It is advantageous for holding the internal data structure. For example, 3 * 3 convolution with holes, 5 * 5 convolutions with holes, etc. can be used as candidate convolution operations.

As an example, for one confirmed waiting layer in the CNN model 21 shown in FIG. 3, the candidate pooling operation may include a 1 * n pooling operation, where n is a predetermined natural number of 2 or more. The purpose of using the 1 * n pooling operation of a rectangular window is to make the output of the CNN model 21 fit the "long" and "narrow" features of the text sequence. Therefore, the final waiting layer of the CNN model 21 preferably uses a rectangular window instead of the traditional square window when performing the pooling operation, thereby reducing the width of the mapped features and finally. It guarantees that the feature sequence is long and narrow, and can match the recognition of the text sequence of the subsequent RNN model 22. For example, 1 * 2 maximum pooling or average pooling, 1 * 3 maximum pooling or average pooling, and the like can be used as candidate pooling operations.

In one example, multiple candidate convolution and candidate pooling operations contained for one determined waiting layer of CNN model 21 in the predefined candidate model space, as shown in Figure 2, are of different types and / or different sizes. Candidate convolution operations and / or candidate pooling operations of different types and / or different sizes may be included. For example, the candidate convolution operation described above includes conventional 3 * 3 convolution, conventional 5 * 5 convolution, separable 3 * 3 convolution, separable 5 * 5 convolution, and perforated 3 * 3 Operations such as convolution and 5 * 5 convolution with holes may be included, the candidate pooling operations described above are 1 * 2 maximum pooling, 1 * 2 average pooling, 1 * 3 maximum pooling, 1 * 3 maximum pooling. Operations such as average pooling may be included.

In one preferred embodiment, in the default convolutional neural network model, the inputs and outputs of one fixed-waiting layer may each be connected to an additional convolutional layer to achieve a 1 * 1 convolutional operation (1 * 1 convolutional operation). In other words, the additional convolutional layer is preset and is not determined by search). Similar to applying a 1 * 1 convolution operation before and after the convolution layer in the Residual Network, in this preferred embodiment, 1 * 1 * 1 is connected to the inputs and outputs of the waiting layer, respectively. By using an additional convolution layer to achieve the convolution operation of 1, it is possible to reduce the number of input and output channels of the confirmation waiting layer, thereby reducing the complexity of the model due to the reduction of parameters. It is advantageous for reduction.

As an option, in one example, the additional convolution layer that realizes the 1 * 1 convolution operation is connected to the decision-waiting layer by the nonlinear activation function, which is advantageous for increasing the nonlinear characteristics of the network. FIG. 4 shows an example of one deterministic waiting layer in the CNN model 21 in FIG. 3 (for example, the i-th layer in the CNN model 21, i = 1, 2, 3, ..., N). The inputs and outputs of 201 are connected by the batch normalization layer (BN) and the ramp function (ReLU) to the additional convolutional layers 202a and 202b for realizing the 1 * 1 convolutional operation, respectively.

In one preferred embodiment, for one definite waiting layer of a convolutional neural network model, in addition to the plurality of candidate convolution and candidate pooling operations described above, the predefined candidate model space is further correspondingly said to that layer. Can include candidate connections from to each subsequent decision-waiting layer in the convolutional neural network model. In addition, by performing search and joint training in the candidate connection for one definite waiting layer of the convolutional neural network model, at least one connection from that layer to the subsequent definite waiting layer in the convolutional neural network model is performed. Can also be confirmed.

In the example of FIG. 3, candidate connections from the first layer and the second layer in the CNN model 21 to the subsequent confirmation waiting layer are shown exemplarily. Regarding the i-th layer (i = 1, 2, 3, ..., N-2) as the current confirmation waiting layer in the CNN model 21, the connection from that layer to the i + 1 layer in the CNN model 21 is generally made. In addition to suspending, it is further determined by search and joint training whether each candidate connection from that layer to the i + 2 layer, i + 3 layer, ..., N layer in the CNN model 21 should be suspended. To do. By deferring the appropriate candidate connections, a good balance between the processing power and processing load of the CNN model can be achieved.

In the above, based on FIGS. 3 and 4, the specific structure of the convolutional neural network model and the search performed for the confirmation waiting layer of the convolutional neural network model in the predefined candidate model space have been described. Then, returning to FIG. 3 and referring to FIG. 5, the concrete structure of the recurrent neural network model and the confirmation waiting layer of the recurrent neural network model in the predefined candidate model space are performed. Explain the search.

As shown in FIG. 3, the RNN model 22 may include L awaiting layers, and search and joint training can determine the structure and optimization parameters of each layer of the L layers. ..

In one preferred embodiment, the initial recurrent neural network model may have a predetermined number of nodes for each decision-waiting layer, and the predefined candidate model space is each decision-waiting for the recurrent neural network model. A plurality of candidate activation functions may be included for each layer, and among these, each layer waiting for confirmation of the recurrent neural network model can be searched and jointly trained in the plurality of candidate activation functions. The activation function of the node is determined, and the connection relationship between each node is also determined.

FIG. 5 is a diagram for explaining one fixed waiting layer in the RNN model shown in FIG. It shows the possible structure of one fixed waiting layer in the RNN model. As shown in FIG. 5, a total of 6 nodes from node 1 to node 6 are set in the confirmation waiting layer. A plurality of candidate activation functions are set for each node k, and may include, for example, a tanh (tangent) function, a ReLU function, an identity (identity) function, and a Sigmoid function, and from node k to node k. Candidate connections to each subsequent node such as +1 and node k + 2, ..., Node 6 (k = 1, 2, ..., 6) are also set. By performing search and joint training, the activation function of each node and the connection relationship between each node can be determined. In other words, the search and joint training can determine the activation function of each node k, and which (one or more) of all the connections shown in FIG. 5 is reserved. Can be confirmed.

Since the recurrent neural network generally uses the same structure in each layer, the RNN model 22 is used when searching and / or training the confirmation waiting layer of the RNN model 22 based on FIG. 5 above. For each layer of, the same structure may be searched and the same parameters may be used.

As described above, based on FIGS. 3 to 5, an example of a specific structure of an overall model including a convolutional neural network model and a recurrent neural network model connected to each other, and an example of a search operation for obtaining the overall model. Explained. Based on this example, with reference to FIG. 6, one processing example for obtaining an optimized model by realizing search and end-to-end training will be described.

FIG. 6 shows an exemplary flow 600 of the process of acquiring an optimized model. As shown in FIG. 6, the exemplary flow 600 may include the following steps:

Step S601: An initial convolutional neural network model having a fixed waiting layer and an early recurrent neural network model having a fixed waiting layer are connected to each other by searching for each fixed waiting layer in a predefined candidate model space. Obtain the preferred overall architecture of the convolutional neural network model and the recurrent neural network model; and Step S603: By end-to-end joint training on the preferred overall architecture of the convolutional neural network model and the recurrent neural network model connected to each other. Obtain the preferred overall architecture optimization parameters.

In the exemplary flow 600, steps S601 and S603 realize optimization of the structure of the entire model and optimization of specific parameters, respectively. By optimizing the architecture of the model and optimizing the parameters of the model under the optimal architecture, the exemplary flow 600 can simplify the process. However, based on the contents of the present invention, those skilled in the art should understand that, in theory, the architecture of the model and the optimization of the parameters of the model can be simultaneously realized if there is sufficient computing power. Thus, in the context of this specification, the specific operation, structure and / or connection of the confirmed waiting layer "determined by search and joint training" can actually be realized by the process of step S601. , And the corresponding optimization parameters can be obtained by the processing of subsequent step S603.

Preferably, the optimization of the structure of the whole model in step S601 and the optimization of the parameters in step S603 are realized by a unified standard. That is, steps S601 and S603 are all their own with the optimization goals of a loss function indicating the probability of accurately recognizing the text and minimizing the overall complexity of the convolutional neural network model and the recurrent neural network model. Processing can be performed.

As an example, in the search process of step S601, for each layer in the CNN model 21 as shown in FIG. 3, the operation of the layer is searched for in a plurality of candidate convolution operations and candidate pooling operations, and the operation of the layer is selectively selected. It searches for at least one of the candidate connections from the layer to each subsequent confirmed waiting layer in the CNN model 21, and multiple candidate actives for each confirmed waiting layer in the RNN model 22 shown in FIG. The activation function of each node in the layer can be searched in the conversion function, and the connection relationship between each node can be determined.

For each possible overall structure of the interconnected CNN model 21 and RNN model 22 obtained by the above search (eg, the overall CRNN structure shown in FIG. 3), the whole, under the initial or random parameters of the structure. It is possible to calculate the loss function, which indicates the probability that the structure will recognize the text accurately, and the overall complexity of the convolutional neural network model and the recurrent neural network model. In other words, the search process for the preferred structure in step S601 does not change the parameter values for each possible overall structure, but uses, for example, a randomly determined initial parameter value. Calculate the above-mentioned loss function and overall complexity of each overall structure of all possible overall structures of the interconnected CNN model 21 and RNN model 22 obtained by the search, and both are all given A structure that satisfies the requirements can be selected as the preferred architecture for the overall model determined in step S601.

For example, the overall complexity threshold is set in advance, and the loss function expression (performance) indicating the probability of accurately recognizing the text is best possible from among the possible structures in which the overall complexity is smaller than the threshold. The structure can be selected as the preferred architecture for the overall model determined in step S601. Alternatively, the preferred architecture of the overall model may be determined by directly constructing the overall optimization function that simultaneously represents the loss function and the overall complexity described above, and finding the solution of the optimization function.

As an example, given an input image containing a text sequence with a real label, the CTC layer 23 in FIG. 3 outputs the exact sequence label for the entire model m with the basic architecture shown in FIG. Using probabilities, we construct a loss function, such as the output of CTC layer 23, as a loss function LOSS (m), which indicates the probability that the overall model will accurately recognize the text, as shown in formula (1) below. be able to.

Here, p (z | x) represents the probability of giving the input x and outputting the sequence z, and S is a training collection. The loss function of formula (1) is to take a negative logarithmic value with respect to the product of the probabilities of outputting an accurate label after giving a sample, and the smaller the loss function, for example, as shown in FIG. There is a high probability that the overall model shown will recognize the text sequence accurately.

It is also possible to calculate the overall complexity of the overall model m constructed based on the CNN model 21, RNN model 22, and CTC layer 23 by various conventional methods. For example, in this example, since the computational complexity of the convolutional operation of the CNN model 21 determines the computational complexity of the entire model m, the computational complexity of the convolutional operation FLOP (m) in the CNN model 21 represents the overall complexity of the entire model m. be able to.

である。 As an example, the number of input channels of the i-th convolutional layer in the CNN model 21 is C _in , the number of output channels is C _out , and the scale of convolution is w * w (w is a natural number). ), And the size of the feature map is H * W (H and W are natural numbers), the calculated amount of the convolutional layer is

Is.

If the convolution operation of the relevant convolution layer is a separable convolution, the complexity of the convolution layer can be modified to the following formula (2').

Based on the computational complexity of each convolution layer obtained by the above formula (2) or (2'), the sum of the computational complexity of all convolution layers is calculated as in the following formula (3). The overall complexity FLOP (m) of the model m can be obtained.

In one example, based on the loss function LOSS (m) and the total complexity FLOP (m) obtained by the above method, a total optimization function such as the following formula (4) can be constructed.

In formula (4), FLOP ₀ and ω are preset constants based on specific applications. In one example, FLOP ₀ may be set to the target complexity of the overall model and ω may be set to 0.07.

Preferably, by the reinforcement learning method, the goal is for the global optimization function as in formula (4) to take the minimum value, for example, each fixed waiting layer and RNN in the CNN model 21 as shown in FIG. An optimal overall model architecture can be determined by performing an iteratively search for each confirmation waiting layer in model 22. In addition, based on the explanation of the content of the present invention, those skilled in the art can use various conventional optimization methods, for example, reinforcement learning, to perform the overall optimization function as described in the above formula (4). By finding the optimal solution of the optimization function, the optimized overall model architecture can be determined.

In the preferred embodiment described above, instead of just representing the accuracy of the prediction, the loss function, which represents the probability that the overall model will recognize the text accurately, is enhanced by using it as part of the optimization goal. It is advantageous to establish a better system structure by learning. In addition, by adopting overall complexity based on computational complexity rather than platform inference time as another part of the optimization goal, it is not affected by the platform that uses the model in the optimization process, and It is also advantageous to transfer the model trained on one platform to another platform.

Of course, based on the content of the present invention, those skilled in the art should understand that, depending on the specific application or task, the accuracy rate ACC (m) indicating the accuracy of the prediction is lost in the probability of accurately recognizing the text. Use in place of the function LOSS (m) and / or use the delay time T (m) of the whole model instead of the total complexity FLOP (m) of convolutional neural network models and recurrent neural network models. You may build different global optimization functions or optimization goals. For example, as an alternative, you may get a global optimization function such as the formula (4') below.

In formula (4'), T ₀ and ω are preset constants based on specific applications. In one example, T ₀ may be set to the target value for the expected delay time of the overall model and ω may be set to 0.07.

One alternative optimization method does not use the global optimization function such as formula (4) or (4') above, but directly optimizes the loss function such as formula (2) first. By setting the goal and, for example, the overall complexity as in formula (3) as the second optimization goal, and by a method for solving a multi-target optimization problem such as the Pareto best solution algorithm, 2 It is also possible to determine the optimized overall architecture by determining the best solution for one optimization goal. In addition, when two optimization goals are given based on the contents of the present invention, those skilled in the art may adopt a method for solving various multi-goal optimization problems to determine the best solution. It is good, but the detailed description is omitted here.

The process for optimizing the overall model architecture executed in step S601 has been described above. After obtaining the optimization architecture of the entire model by the processing in step S601, in step S603, using the same optimization function or optimization goal, the optimization architecture is concretely trained by end-to-end joint training. Parameter optimization can be realized.

Specifically, for example, the CNN model 21 in which the operation and connection of each confirmation waiting layer are already confirmed with the goal of obtaining the minimum value by the global optimization function such as the formula (4), and each confirmation waiting layer. For the RNN model 22 (and the preferred CTC layer 23 connected to the RNN model 22) whose layer structure has already been determined, joint training is performed on the entire model in an end-to-end manner, and each parameter in the entire model is performed. You can get the highest value of. If the architecture of the model is fixed and a global optimization function is given, those skilled in the art may implement such a training process by various conventional methods. The detailed description will be omitted.

As described above, based on the flowchart shown in FIG. 6, while referring to the specific examples shown in FIGS. 3 to 5, for example, exemplary processing for obtaining the overall model architecture used in the image processing method shown in FIG. Explained. By using the exemplary process shown in Figure 6, manual intervention can be reduced and the architecture of the model with good performance can be obtained. Also, the corresponding optimization model can be easily obtained for different tasks and / or data collections in order to obtain the model optimization architecture by searching rather than by completely artificial (manual) design. That is, the model acquired by this method can be easily diverted to different tasks and / or numerical data.

In addition, by optimizing the probability of accurately recognizing the text and performing search and joint training by a multi-target optimization method of optimizing the overall complexity of the model, the exemplary processing shown in FIG. 6 is performed. The acquired model can simultaneously consider the accuracy of recognition and the scale or complexity of the model. Compared to a general recognition model that considers only recognition accuracy, the amount of calculation or scale of the model acquired here that also has optimized overall complexity is relatively small, so there are mobile platform and resource constraints. Especially applicable to the environment.

According to another aspect of the present invention, an image processing device is provided. FIG. 7 is a block diagram showing one exemplary structure of the image processing apparatus according to the embodiment of the present invention.

As shown in FIG. 7, the image processing apparatus 700 may include the following.

Feature extraction unit 701: Inputs awaiting image containing text into a convolutional neural network (CNN) model for character recognition and extracts text features; and text recognition unit 702: Convolutional neural network of extracted text features. Input to a recurrent neural network (RNN) model for word meaning recognition connected to the network model, and recognize the text in the waiting image.

Here, the convolutional neural network (CNN) model used for the feature extraction unit 701 and the recurrent neural network (RNN) model used for the text recognition unit 702 are initial convolutional neural networks having a fixed waiting layer connected to each other. Based on a network model and an early recurrent neural network model with a fixed waiting layer, a search is performed for each fixed waiting layer in a predefined candidate model space, and a convolutional neural network connected to each other in an end-to-end manner. Obtained by performing joint training on network models and recurrent neural network models.

Since the above-mentioned image processing apparatus and its unit can, for example, perform the image processing method described with reference to FIG. 1 and the operation and / or processing of each step thereof, and realize similar effects, the present invention is described here. Then, the detailed explanation is omitted. Further, the CNN model and the RNN model used in the above-mentioned image processing apparatus can be obtained by the exemplary processing for optimizing the model architecture described with reference to FIG. 6 above.

According to another aspect of the present invention, an information processing device is provided. The information processing apparatus can realize the image processing method according to the embodiment of the present invention, which includes a processor, and the processor is configured as follows, that is, a processing waiting image including text. To the convolutional neural network model for character recognition and extract the text features; and to connect the extracted text features to the recurrent neural network model for word meaning recognition connected to the convolutional neural network model. Input and recognize the text in the image waiting to be processed. Here, the convolutional neural network model and the recurrent neural network model are predefined candidates based on the initial convolutional neural network model having a definite waiting layer and the initial recurrent neural network model having a definite waiting layer connected to each other. It is obtained by searching each fixed waiting layer in the model space and performing joint training on convolutional neural network models and recurrent neural network models connected to each other in an end-to-end manner.

The processor of the information processing device may be configured to realize similar effects by, for example, performing the image processing method described with reference to FIG. 1 and the operation and / or processing of each step thereof. , Here, the detailed description is omitted. In addition, the CNN model and RNN model used by the processor may be obtained by exemplary processing for optimizing the architecture of the model described with reference to FIG. 6 above.

Optionally, the processor may be configured as follows: a loss function that indicates the probability of accurately recognizing the text, and the minimum overall complexity of the convolutional neural network model and the recurrent neural network model. Search and joint training are performed with optimization as the optimization target, and the convolutional neural network model and recurrent neural network model after optimization are acquired.

As an example, the predefined candidate model space may include multiple candidate convolution and candidate pooling operations for one convolutional neural network model decision waiting layer, and the processor may be configured as follows: That is, for one confirmation waiting layer of the convolutional neural network model, the operation of the layer is confirmed by performing search and joint training in a plurality of candidate convolution operations and candidate pooling operations.

As an example, the predefined candidate model space may include for one decision-waiting layer of the convolutional neural network model and further candidate connections from that layer to each subsequent decision-waiting layer in the convolutional neural network model. Further, it may be configured as follows, that is, one confirmation waiting layer of the convolutional neural network model is searched and jointly trained in the candidate connection, and the subsequent layer in the convolutional neural network model is followed. Confirm at least one connection to the confirmation waiting layer.

As an option, in the convolutional neural network model, the inputs and outputs of one definite waiting layer are each connected to an additional convolutional layer to achieve a 1 * 1 convolutional operation.

Optionally, the candidate convolution operation includes a perforated convolution operation.

Optionally, the candidate pooling operation includes a 1 * n pooling operation, where n is a predetermined natural number greater than or equal to 2.

Optionally, the plurality of candidate convolution and / or candidate pooling operations includes different types and / or different size candidate convolution operations and / or different types and / or different size candidate pooling operations.

As an example, the initial recurrent neural network model has a predetermined number of nodes for each fixed waiting layer, of which the predefined candidate model space is a plurality of candidates for each fixed waiting layer of the recurrent neural network model. Including an activation function, the processor may further be configured as follows: for each recurrent neural network model waiting layer, search and joint training among multiple candidate activation functions. The activation function of each node in the layer is determined, and the connection relationship between the nodes is also determined.

FIG. 8 is a structural diagram of a hardware configuration (general-purpose machine) 800 that can realize the information processing method and device according to the embodiment of the present invention.

The general-purpose machine 800 may be, for example, a computer system. The general-purpose machine 800 is merely an example, and does not limit the application range or function of the method and device according to the present invention. Further, the general-purpose machine 800 does not depend on any module, assembly, or a combination thereof in the above-mentioned method and device.

In FIG. 8, the central processing unit (CPU) 801 performs various processes based on the program stored in the ROM 802 or the program rodged from the storage unit 808 to the RAM 803. The RAM 803 can also store data required when the CPU 801 performs various processes according to needs. CPU 801 and ROM 802 and RAM 803 are connected to each other via bus 804. The input / output interface 805 is also connected to bus 804.

Further, the following components are connected to the input / output interface 805, that is, an input unit 806 including a keyboard and the like, an output unit including a display such as a liquid crystal display (LCD), and a speaker. The 807, the storage unit 808 including the hard disk, and the communication unit 809 including the network interface card, for example, the LAN card and the modem. The communication unit 809 performs communication processing via a network such as the Internet or LAN. Drive 810 may be connected to input / output interface 805, if desired. The removable medium 811 (for example, a semiconductor memory) is set in the drive 810 as needed, and the computer program read from the medium 811 can be installed in the storage unit 808.

The present invention also provides a program product that includes a machine-readable command code. When such a command code is read and executed by the machine, the method according to the embodiment of the present invention described above can be executed. Correspondingly, carry such program products, such as magnetic disks (including floppy disks (registered trademarks)), optical disks (including CD-ROMs and DVDs), magneto-optical disks (MD (registered trademarks)). ), And various storage media such as semiconductor storage devices are also included in the present invention.

The above-mentioned storage medium may include, but is not limited to, for example, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor storage device, and the like.

In addition, each operation (process) in the above method can also be realized by a method of a computer-executable program stored in various machine-readable storage media.

In addition, the above examples and the like will be further disclosed as additional notes as follows.

(Appendix 1)
It is an image processing method
Input awaiting image containing text into a convolutional neural network model for character recognition, extract text features; and recursive type for word recognition to connect the extracted text features with a convolutional neural network model. Includes inputting into a neural network model and recognizing text in awaiting images
The convolutional neural network model and the recurrent neural network model are in a predefined candidate model space based on the initial convolutional neural network model having a definite waiting layer and the initial recurrent neural network model having a definite waiting layer connected to each other. A method obtained by searching for each fixed waiting layer in, and performing joint training on convolutional neural network models and recurrent neural network models connected to each other in an end-to-end manner.

(Appendix 2)
The image processing method described in Appendix 1
After optimization, search and joint training are performed with the optimization goal of minimizing the overall complexity of the convolutional neural network model and the recurrent neural network model, as well as the loss function that indicates the probability of accurately recognizing the text. A method of obtaining a convolutional neural network model and a recurrent neural network model.

(Appendix 3)
The image processing method described in Appendix 1 or 2.
The predefined candidate model space contains multiple candidate convolution and candidate pooling operations for one definite waiting layer of the convolutional neural network model.
A method of confirming the operation of one confirmation waiting layer of a convolutional neural network model by performing search and joint training in a plurality of candidate convolution operations and candidate pooling operations.

(Appendix 4)
The image processing method described in Appendix 3
The predefined candidate model space includes for one deterministic waiting layer of the convolutional neural network model and further containing candidate connections from that layer to each subsequent deterministic waiting layer in the convolutional neural network model.
By performing a search and joint training in the candidate connection for one definite waiting layer of the convolutional neural network model, at least one connection from the layer to the subsequent definite waiting layer in the convolutional neural network model is acquired. Method.

(Appendix 5)
The image processing method described in Appendix 3 or 4,
In a convolutional neural network model, the input and output of one fixed-waiting layer are each connected to an additional convolutional layer to achieve a 1 * 1 convolutional operation.

(Appendix 6)
The image processing method described in Appendix 3
Candidate convolution operations include perforated convolution operations.
And / or candidate pooling operations include 1 * n pooling operations, where n is a predetermined natural number greater than or equal to 2.

(Appendix 7)
The image processing method described in Appendix 3
Multiple candidate convolution operations and candidate pooling operations
Candidate convolution operations of different types and / or different sizes; and / or methods that include candidate pooling operations of different types and / or different sizes.

(Appendix 8)
The image processing method described in Appendix 1 or 2.
The initial recurrent neural network model has a predetermined number of nodes for each decision waiting layer.
The predefined candidate model space contains multiple candidate activation functions for each decision-waiting layer of the recurrent neural network model.
By performing search and joint training in a plurality of candidate activation functions for each confirmation waiting layer of the recurrent neural network model, the activation function of each node in the layer and the connection relationship between each node are determined. Method.

(Appendix 9)
An information processing device that includes a processor
The processor is
Input awaiting image containing text into a convolutional neural network model for character recognition, extract text features; and recursive type for word recognition to connect the extracted text features with a convolutional neural network model. It is configured to input into a neural network model and recognize text in awaiting images.
The convolutional neural network model and the recurrent neural network model are in a predefined candidate model space based on the initial convolutional neural network model having a definite waiting layer and the initial recurrent neural network model having a definite waiting layer connected to each other. A device that is acquired by searching for each fixed waiting layer in, and performing joint training on convolutional neural network models and recurrent neural network models that are connected to each other in an end-to-end manner.

(Appendix 10)
The information processing device described in Appendix 9
The processor is also
After optimization, search and joint training are performed with the optimization goal of the loss function showing the probability of accurately recognizing the text and the minimization of the overall complexity of the convolutional neural network model and the recurrent neural network model. A device configured to obtain a convolutional neural network model and a recurrent neural network model.

(Appendix 11)
The information processing device according to Appendix 9 or 10.
The predefined candidate model space contains multiple candidate convolution and candidate pooling operations for one definite waiting layer of the convolutional neural network model.
The processor is also
A device configured to perform search and joint training in a plurality of candidate convolution operations and candidate pooling operations for one confirmation waiting layer of a convolutional neural network model, and to confirm the operation of the layer.

(Appendix 12)
The information processing device described in Appendix 11
The predefined candidate model space includes for one deterministic waiting layer of the convolutional neural network model and further containing candidate connections from that layer to each subsequent deterministic waiting layer in the convolutional neural network model.
The processor is also
It is configured to search and joint train one of the convolutional neural network models in the candidate connection to determine at least one connection from that layer to the subsequent definite waiting layer in the convolutional neural network model. The device to be done.

(Appendix 13)
The information processing device according to Appendix 11 or 12.
In a convolutional neural network model, a device in which the inputs and outputs of one definite waiting layer are each connected to an additional convolutional layer to achieve a 1 * 1 convolutional operation.

(Appendix 14)
The information processing device described in Appendix 11
Candidate convolution operations include perforated convolution operations.
And / or
Candidate pooling operations include 1 * n pooling operations, where n is a predetermined natural number greater than or equal to 2.

(Appendix 15)
The information processing device described in Appendix 11
Multiple candidate convolution operations and candidate pooling operations
A device that includes candidate convolution operations of different types and / or different sizes; and / or candidate pooling operations of different types and / or different sizes.

(Appendix 16)
The information processing device according to Appendix 9 or 10.
The initial recurrent neural network model has a predetermined number of nodes for each decision waiting layer.
The predefined candidate model space contains multiple candidate activation functions for each decision-waiting layer of the recurrent neural network model.
The processor is also
For each confirmation waiting layer of the recurrent neural network model, the activation function of each node in the layer and the connection relationship between each node are determined by performing search and joint training in a plurality of candidate activation functions. A device that is configured in.

(Appendix 17)
A storage medium that stores a machine-readable command code.
When the command code is read and executed by the machine, the machine is made to execute the image processing method, and the image processing method is
Input awaiting image containing text into a convolutional neural network model for character recognition, extract text features; and recursive type for word recognition to connect the extracted text features with a convolutional neural network model. Includes inputting into a neural network model and recognizing text in awaiting images
The convolutional neural network model and the recurrent neural network model are in a predefined candidate model space based on the initial convolutional neural network model having a definite waiting layer and the initial recurrent neural network model having a definite waiting layer connected to each other. A storage medium obtained by searching for each fixed waiting layer and performing joint training on convolutional neural network models and recurrent neural network models connected to each other in an end-to-end manner.

Although the preferred embodiment of the present invention has been described above, the present invention is not limited to this embodiment, and any modification to the present invention belongs to the technical scope of the present invention unless the gist of the present invention is departed.

Claims (10)

It is an image processing method
Input awaiting image containing text into a convolutional neural network model for character recognition, extract text features; and recursive type for word recognition to connect the extracted text features with a convolutional neural network model. Includes inputting into a neural network model and recognizing text in awaiting images
The convolutional neural network model and the recurrent neural network model are in a predefined candidate model space based on the initial convolutional neural network model having a definite waiting layer and the initial recurrent neural network model having a definite waiting layer connected to each other. A method obtained by searching for each fixed waiting layer in, and performing joint training on convolutional neural network models and recurrent neural network models connected to each other in an end-to-end manner. The image processing method according to claim 1.
After optimization, search and joint training are performed with the optimization goal of minimizing the overall complexity of the convolutional neural network model and the recurrent neural network model, as well as the loss function that indicates the probability of accurately recognizing the text. A method of obtaining a convolutional neural network model and a recurrent neural network model. The image processing method according to claim 1 or 2.
The predefined candidate model space contains multiple candidate convolution and candidate pooling operations for one definite waiting layer of the convolutional neural network model.
A method of confirming the operation of one confirmation waiting layer of a convolutional neural network model by performing search and joint training in a plurality of candidate convolution operations and candidate pooling operations. The image processing method according to claim 3.
The predefined candidate model space includes for one deterministic waiting layer of the convolutional neural network model and further containing candidate connections from that layer to each subsequent deterministic waiting layer in the convolutional neural network model.
By performing a search and joint training in the candidate connection for one definite waiting layer of the convolutional neural network model, at least one connection from the layer to the subsequent definite waiting layer in the convolutional neural network model is acquired. Method. The image processing method according to claim 3.
In a convolutional neural network model, the input and output of one fixed-waiting layer are each connected to an additional convolutional layer to achieve a 1 * 1 convolutional operation. The image processing method according to claim 3.
Candidate convolution operations include perforated convolution operations; and / or candidate pooling operations include 1 * n pooling operations, where n is a predetermined natural number greater than or equal to 2. The image processing method according to claim 3.
Multiple candidate convolution operations and candidate pooling operations
Candidate convolution operations of different types and / or different sizes; and / or methods that include candidate pooling operations of different types and / or different sizes. The image processing method according to claim 1 or 2.
The initial recurrent neural network model has a predetermined number of nodes for each decision waiting layer.
The predefined candidate model space contains multiple candidate activation functions for each decision-waiting layer of the recurrent neural network model.
By performing search and joint training in a plurality of candidate activation functions for each confirmation waiting layer of the recurrent neural network model, the activation function of each node in the layer and the connection relationship between each node are determined. Method. An information processing device that includes a processor
The processor
Input awaiting image containing text into a convolutional neural network model for character recognition, extract text features; and recursive type for word recognition to connect the extracted text features with a convolutional neural network model. It is configured to input into a neural network model and recognize text in awaiting images.
The convolutional neural network model and the recurrent neural network model are in a predefined candidate model space based on the initial convolutional neural network model having a definite waiting layer and the initial recurrent neural network model having a definite waiting layer connected to each other. A device that is acquired by searching for each fixed waiting layer in, and performing joint training on convolutional neural network models and recurrent neural network models that are connected to each other in an end-to-end manner. A storage medium in which a program for executing the image processing method according to any one of claims 1 to 8 is stored.

Images