JP4013489B2 - Corresponding category search system and method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a technique for determining a category correspondence between a plurality of category structures respectively constructed for different languages, targeting a category structure in which a document set is classified into a plurality of categories.
[0002]
[Prior art]
One method for facilitating access to a large amount of document collections is a method of classifying the document collections into a plurality of categories. When the document set is classified into categories, it is possible to efficiently obtain a desired document by performing a search for only a category to which a document desired by a user is supposed to belong. In some cases, a set of documents may be manually classified into categories, or the classification as described in the document “Prospects for Information Retrieval Cognitive Approach, by David Ellis, Maruzen Co., Ltd. (1994)” is automated. Many techniques have been proposed.
[0003]
When such a categorized document set (hereinafter also referred to as a category structure) is constructed for a plurality of languages, a category correspondence between the plurality of category structures (including a document set having similar semantic contents) Determining the correspondence between categories) is important in performing a document search across multiple languages (multilingual document search). In other words, by limiting the document set of the target language (language used for secondary search) as the search target to categories close to the search request in the source language (language used for direct search), the accuracy of the search Can be improved.
[0004]
As a method for automatically determining the correspondence between such categories, a method using a multilingual document search method is mainly used. For example, the literature "Hiroshi Masuichi, Raymond Flournoy, Stefan Kaufmann and Stanley Peters," Query Translation Method for Cross Language Information Retrieval ", The Proceedingsof Machine Translation Summit VII '99 Workshop on Machine Translation for Cross Language Information Retrieval, (1999)" In Using a set of translation pairs (parallel corpus) as learning data, each document written in a different language is represented as a document vector in the same vector space, and the remainder between vectors Technique for multilingual document retrieval has been proposed a value as a similarity between documents. By using this method, a sum of document vectors corresponding to all documents belonging to a category is defined as a category vector, and a cosine between category vectors is defined as a similarity, whereby a plurality of languages constructed for different languages are defined. It becomes possible to determine the correspondence of categories between category structures.
[0005]
[Problems to be solved by the invention]
However, at present, it is difficult to say that a practically sufficient accuracy is obtained in determining the corresponding category by the above method. In general, the biggest factor that reduces the search accuracy of multilingual information search is a problem of meaning ambiguity of words or phrases. When a certain word (phrase) in the first language is translated into a word (phrase) in the second language, there are many translation candidates. For example, the word “base” in English is “base” as a military term, “塁” as a baseball term, “supporting body” as a political term, “base” as a mathematical term, and “base” as a chemical term. There are various translation candidates depending on the field, such as “term” as a grammatical term and “main component of (paint)” as an architectural term. Since these translation candidates are often field-dependent, it is said that high accuracy can be obtained in multilingual information search if the search target is limited to a set of documents in a specific field. That is, there are translated words in the category corresponding to the field of the category, and it is necessary to perform a multilingual document search using an appropriate translated word for each field. In the method using the document vector described above, a certain parallel corpus is used as learning data, so that an appropriate document vector corresponding to the field cannot be generated, and therefore the problem of semantic ambiguity cannot be solved. If a parallel corpus can be prepared for each category, it is possible to generate an appropriate document vector according to the field, but generally it is difficult to obtain a parallel corpus, and in fact, such an approach is impossible. is there. In addition to the multilingual document search method using parallel corpus as learning data, many multilingual document search methods using bilingual dictionaries have been proposed. However, if a general bilingual dictionary is used, meaning ambiguity It is completely the same that the above problem cannot be solved and it is practically impossible to prepare a bilingual dictionary for each field (category).
[0006]
The present invention has been made in view of these points, and an object thereof is to provide a system that can determine the correspondence between categories with high accuracy without preparing a parallel corpus for each category. .
[0007]
[Means for Solving the Problems]
Literature "Hiroshi Masuichi, Raymond Flournoy, Stefan Kaufmann and Stanley Peters," A Bootstrapping method for Extracting Bilingual Text Pairs ", The Proceedings of The 18th International Conference on Computational Linguistics, pp.1066-1070 (2000)" in, as follows: A similar bilingual similar document pair determination method has been proposed.
[0008]
“Search a multilingual document using a parallel corpus as initial learning data, determine a similar bilingual document pair from a set of documents in which bilingual documents are mixed, and use the obtained document pair as initial learning data. The multilingual document search is performed again based on the obtained learning data, and the learning pair is obtained by alternately repeating the multilingual document searching process and the process of adding the obtained document pair to the learning data. Growing up and finally getting a highly accurate parallel corpus (a bilingual document pair). "
[0009]
As described in the above document, this method works only when the semantic contents of each document in the document set that is the target of multilingual document search are similar (in the same field). The present invention takes advantage of this property of the above approach. That is, the above-described method is performed by using a combination of a first category including a document set written in a first language and a second category including a document set written in a second language as a target of multilingual document search. When the learning pair grows, it is determined that the fields of the first category and the second category are similar.
[0010]
As shown in FIG. 1, one configuration of the present invention is a category that holds a first category structure generated for a first language and a second category structure generated for a second language. The category holding unit uses the structure holding unit (1), the learning data holding unit (2) that holds learning data when performing multilingual document search, and the learning data held in the learning data holding unit. A multilingual document search is performed for the category in the first category and the category in the second category that are held, and a bilingual document pair of the similar first language and second language is determined. Language document search means (3), a document result obtained by multilingual document search, a search result holding means (4) for adding the document pair to the learning data holding means, and a search result holding means Category correspondence determining means (5) for determining correspondence between categories with reference to a document pair, and in this configuration, multilingual search processing by the multilingual document search means and search result holding The process of adding a document pair to learning data by means is repeated alternately.
[0011]
It should be noted that the present invention can be realized not only as an apparatus or a system but also as a method, and at least a part thereof can be configured as a computer program.
[0012]
One of the above-described configurations of the present invention and other configurations of the present invention are clearly set forth in the appended claims, and are described in detail below using examples.
[0013]
DETAILED DESCRIPTION OF THE INVENTION
Examples of the present invention will be described below.
[0014]
FIG. 2 shows the configuration of the corresponding category search system according to the embodiment of the present invention. In this embodiment, explanation is given for Japanese and English, but the same effect can be obtained in any language as long as morphological analysis processing (processing for dividing a sentence into words) is applicable. Can be obtained.
[0015]
In FIG. 2, the category structure holding means 11 holds a category structure (first category structure and second category structure) in which a plurality of Japanese documents and a plurality of English documents are classified into categories and stored. It is means to do.
[0016]
The learning data holding means 12 is means for holding a set of Japanese-English translation document pairs (Japanese-English parallel corpus) as initial learning data. The parallel corpus is not limited to a particular field, and is a parallel corpus having general contents that are easily available. When a Japanese-English document pair is received from the search result holding means 16, it is added to the parallel corpus as initial learning data and held as a new Japanese-English parallel corpus.
[0017]
The word vector generation means 13 uses, as learning data, a Japanese-English parallel corpus held in the learning data holding means 12, and for all Japanese words and English words contained therein, corresponding multidimensional vectors (word vectors). ). Hereinafter, an algorithm for calculating a word vector will be described.
[0018]
[Step 1]: All Japanese and English documents included in the learning data are subjected to a morphological analysis process.
[Step 2]: Of all the words obtained in Step 1, n words are selected in descending order of appearance frequency in the learning data. The n words obtained here are called feature expression words. The value of n is on the order of thousands.
[Step 3]: A matrix corresponding to all Japanese / English words and feature expression words obtained in Step 1 is created for each row and column. If the total number of different Japanese / English words obtained in step 1 is 100,000 and the value of n is 3,000, a matrix of 100,000 rows × 3,000 columns can be formed. For each element of this matrix, how many times the word corresponding to the row of the element and the feature expression word corresponding to the column co-occur in all Japanese-English translation pairs included in the learning data (simultaneously Record whether it has appeared). That is, the Japanese-English translation pair is regarded as one document, and the number of co-occurrence in the document is counted. The matrix thus obtained is called a co-occurrence matrix. In this way, it is possible to create a co-occurrence matrix that expresses all Japanese words and all English words by an n-dimensional vector. This vector can be said to be a vector indicating in what context each word tends to appear.
[0019]
[Step 4]: Since the n-dimensional vector obtained in Step 3 has a large number of dimensions, the calculation time becomes enormous in the processing required later. Therefore, in order to limit the calculation process to the real time range, the original n-dimensional vector is compressed into an n′-dimensional (several hundred dimensions) vector by a matrix dimension compression method. There are various dimensional compression methods, but "Berry, M., Do, T., O'Brien, G., Krishna, V. and Varadhan, S.," SVDPACKC USER'S GUIDE ".Tech. Rep. CS-93-194.University of Tennessee, Knoxville, TN (1993) "is a representative example of Single Value Decomposition. The n′-dimensional vectors obtained for all Japanese words and English words in this way are referred to as word vectors.
[0020]
The document vector generation means 14 uses the word vector obtained by the word vector generation means 13 to correspond to all Japanese documents in category A and all English documents in category B held in the category structure holding means 11. A means for calculating a document vector. First, morphological analysis processing is performed on all Japanese documents in category A and all English documents in category B held in category structure holding means 11 and divided into words. Next, a vector obtained by normalizing the sum of the word vectors corresponding to all the words included in each document (with the vector length set to 1) is calculated, and the obtained vector is set as the document vector. However, words whose corresponding word vectors are not generated by the word vector generation means 13 are ignored.
[0021]
The multilingual search means 15 is a category pair of an arbitrary Japanese category (category A) in the first category and an arbitrary English category (category B) in the second category held in the category structure holding means 11. This is a means for searching for similar Japanese-English document pairs. Therefore, it is assumed that the following processing is performed for each category pair of any Japanese category and English category (any combination of category A and category B).
[0022]
First, by referring to the document vector obtained from the document vector generation means 14, pairs of Japanese documents and English documents satisfying the following conditions are extracted from all document sets belonging to category A and category B.
[0023]
“The document vector corresponding to the Japanese document in the document pair has the highest degree of relevance (the inner product value is large) is the English document vector in the pair, and conversely the most related to the English document vector in the pair. The Japanese document vector with the highest degree is the Japanese document vector in the pair. "
[0024]
Next, out of Japanese-English document pairs satisfying the above conditions, a pair is extracted in which the value of the inner product between the Japanese-English document vectors corresponding to the Japanese-English documents in the pair is larger than a preset threshold value. The Japanese-English document pairs obtained in this way have very close meanings and can be used as learning data.
[0025]
The search result holding means 16 is a means for holding the Japanese-English document pair set obtained from the multilingual search means 15 for the categories A and B in the computer. The obtained document pair set is transferred to the learning data holding means 12 as a part of new learning data.
[0026]
In this way
(1) Based on the learning data held in the learning data holding means 12, the word vector generating means 13 generates a word vector set,
(2) generating a document vector set by the document vector generating means 14;
(3) The multilingual search means 15 extracts Japanese-English document pairs having similar meaning contents,
(4) The search result holding unit 16 adds the obtained document pair to the learning data holding unit 12 as part of the learning data (if already added, replaces the previous one).
By repeating the above process, the search result holding means 16 is used only when the semantic content of the Japanese document set in category A and the English document set in category B are close (category A and category B belong to the same field). The number of document pairs held in will gradually increase.
[0027]
The category correspondence determining unit 17 refers to the ratio of the “total number of document pairs held in the search result holding unit 16” to the “total number of documents included in category A and category B”, and the ratio is determined in advance. If the threshold value T is larger than the threshold value T, it is determined that the category A and the category B are similar (same field) category pairs. In addition, when the above repetition processing is performed a certain number of times or more for category A and category B but the ratio does not exceed the threshold T, category A and category B are similar (in the same field). Determine that it is not a category pair.
[0028]
The determination of the category correspondence may be completed by only one document search. Further, the threshold value may be set in multiple stages. For example, if it is less than the threshold a (a <b) in a predetermined number of document searches, it is determined as non-corresponding. If it is more than b, you may employ | adopt the structure which discriminate | determines that a category respond | corresponds immediately. In short, any configuration may be adopted as long as it is determined that the category corresponds when the document search result indicates a sign that the category correspondence is affirmed.
[0029]
By adopting such a configuration, the category correspondence determining means 17 determines the presence / absence of correspondence for all Japanese category and English category category pairs (any combination of category A and category B). Thus, it is possible to comprehensively determine the correspondence relationship between the first category and the second category.
[0030]
It should be noted that the above-mentioned reference "Hiroshi Masuichi in the present embodiment, Raymond Flournoy, Stefan Kaufmann and Stanley Peters," Query Translation Method for Cross Language Information Retrieval ", The Proceedings of Machine Translation Summit VII'99 Workshopon Machine Translation for Cross Language Information Retrieval , (1999) ", the multilingual document search method based on the vector space method using the parallel corpus as learning data is used, but the search result is repeatedly added to the learning data. If Ukoto multilingual document retrieval method capable of the same effect be any technique is obtained (see FIG. 1).
[0031]
For example, a document written in a first language in category A is translated into a second language by a machine translation system, and a multilingual document search is performed using a document search method for a general single language. The same effect can be obtained by a technique.
[0032]
As an example of realizing a machine translation system using a parallel corpus as learning data, the literature “Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer,“ The Mathematical Sciences: , Computational Linguistics, 32: 263-311, 1993. ".
[0033]
【The invention's effect】
As described above, according to the present invention, the problem of word ambiguity resolution can be avoided without preparing learning data for each field, and the correspondence of categories between multiple category structures constructed for different languages can be obtained. It becomes possible to determine with high accuracy.
[Brief description of the drawings]
FIG. 1 is a diagram showing a configuration of a typical corresponding category search system according to the present invention.
FIG. 2 is a diagram showing a configuration of a corresponding category search system according to an embodiment of the present invention.
[Explanation of symbols]
11 category structure holding means 12 learning data holding means 13 word vector generating means 14 document vector generating means 15 multilingual search means 16 search result holding means 17 category correspondence determining means

Claims (6)

Category structure holding means for holding a first category structure generated for the first language and a second category structure generated for the second language;
Learning data holding means for holding learning data when performing multilingual document search;
Using the learning data held in the learning data holding means, a multilingual document search is performed for the category in the first category structure and the category in the second category structure held in the category structure holding means. A multilingual document search means for determining a bilingual document pair of a similar first language and second language;
Search result holding means for holding a document pair obtained by multilingual document search and adding the document pair to the learning data holding means;
Category correspondence determining means for determining correspondence between categories by referring to document pairs held in the search result holding means;
Corresponding category search system comprising: Category structure holding means for holding a first category structure generated for the first language and a second category structure generated for the second language;
Learning data holding means for holding a translation document pair as learning data when performing multilingual document search;
Using the learning data held in the learning data holding means, the multiple categories based on the vector space method are applied to the categories in the first category structure and the categories in the second category structure held in the category structure holding means. Multilingual document search means for performing a language document search and determining a bilingual document pair of a similar first language and second language;
Search result holding means for holding a document pair obtained by multilingual document search and adding the document pair to the learning data holding means;
Category correspondence determining means for determining correspondence between categories by referring to document pairs held in the search result holding means;
Corresponding category search system comprising: Category structure holding means for holding a first category structure generated for the first language and a second category structure generated for the second language;
Learning data holding means for holding a translation document pair as learning data when performing multilingual document search;
Using the learning data held in the learning data holding means, the document written in the first language belonging to the category in the first category structure held in the category structure holding means is translated into the second language. A document search is performed on the obtained translation document set and a document set belonging to a category in the second category structure, and a bilingual document pair of a similar first language and second language is determined. Language document search means;
Search result holding means for holding a document pair obtained by multilingual document search and adding the document pair to the learning data holding means;
Category correspondence determining means for determining correspondence between categories by referring to document pairs held in the search result holding means;
Corresponding category search system comprising: A category structure holding step for storing a first category structure generated for the first language and a second category structure generated for the second language;
A learning data holding step for storing learning data when performing multilingual document search;
Using the learning data stored in the learning data holding step, performing a multilingual document search for the category in the first category structure and the category in the second category structure stored in the category structure step, A multilingual document search step for determining a bilingual document pair of similar first language and second language;
A search result holding step of holding the document pair obtained in the multilingual document search step and adding the document pair as learning data;
A category correspondence determination step for determining correspondence between categories with reference to the document pair stored in the search result holding step;
A corresponding category search method comprising: A category structure holding step for storing a first category structure generated for the first language and a second category structure generated for the second language;
A learning data holding step for storing learning data when performing multilingual document search;
Using the learning data stored in the learning data holding step, performing a multilingual document search for the category in the first category structure and the category in the second category structure stored in the category structure step, A multilingual document search step for determining a bilingual document pair of similar first language and second language;
A search result holding step of holding the document pair obtained in the multilingual document search step and adding the document pair as learning data;
A category correspondence determination step for determining correspondence between categories with reference to the document pair stored in the search result holding step;
A computer program for searching for a corresponding category, which is used for causing a computer to execute a program. Category structure holding means for holding a first category structure generated for the first language and a second category structure generated for the second language;
Learning data holding means for holding learning data when performing multilingual document search;
Using the learning data held in the learning data holding means, a multilingual document search is performed for the category in the first category structure and the category in the second category structure held in the category structure holding means. A multilingual document search means for determining a bilingual document pair of a similar first language and second language;
Category correspondence determining means for determining correspondence between categories based on document pairs obtained by multilingual document search;
Corresponding category search system comprising:

Images