CN103207921A - Method for automatically extracting terms from Chinese electronic document - Google Patents

Abstract

The invention relates to a method for automatically extracting terms from a Chinese electronic document. The method is characterized by comprising the following steps of: step S01: processing the electronic document into a group of word strings consisting of atomic words with a special property; step S02: counting the frequency of the atomic word strings and substrings, adopting the atomic word string with the appearance times being more than N times as a candidate term, wherein N is a settable parameter; and step S03: deleting the term which only appears as a substring in a candidate term set to obtain a term set appearing in the document, and realizing the purpose for automatically extracting the terms in the Chinese electronic document. The method has the effects and benefits that the real problem and difficulty that the performance for automatically extracting the term is not high and the automation degree is limited can be solved. The high-efficient automatic method for extracting the terms is a foundation for automatically processing a text and can powerfully guarantee the information search, text summarization, content management and the like. The good term extracting method can promote the automation degree and the performance of the work.

Description

A Method of Automatically Extracting Words from Chinese Electronic Documents

technical field

The invention belongs to the field of natural language processing and relates to a method for automatically extracting word sets from Chinese electronic documents.

Background technique

In recent years, with the rapid development of scientific research, economy, and the Internet, the number of electronic documents has increased rapidly. How to quickly and effectively process these massive electronic documents has become one of the key tasks in the fields of information retrieval, knowledge management, and Web services. . As a result, automatic electronic document processing technologies such as text retrieval, classification, and automatic summarization have become research hotspots in related fields. Among these technologies, automatic extraction of all words in electronic documents (referred to as "prompt") is a basic work. The word prompting method of the present invention is aimed at the automatic processing of Chinese electronic documents. If there is no special instruction, the following "documents" all refer to "Chinese electronic documents", and "words" all refer to "Chinese words".

The words (term or word) in the document are divided into two types: atomic words and compound words (also known as compound words). Atomic words (aw) are short words in language that are used to combine to form other new words, and do not follow the principle of meaning combination, such as "system", "knowledge", etc. Compound words (compound words, cw) are content-oriented long words composed of multiple atomic words, and the formation of these words generally follows the principle of meaning combination. For example, "systems engineering", "knowledge management" and so on.

The automatic extraction of atomic words can be easily done based on the atomic word dictionary. Since atomic words are relatively stable and new words rarely appear, they can be extracted based on dictionaries such as Chinese Thesaurus or Chinese Classified Thesaurus, and the accuracy and recall rates are satisfactory.

There are two main types of automatic extraction methods for compound words: one is a method based on statistics, such as a word prompt method based on string frequency and string length. One is a method based on part-of-speech analysis, such as a method of extracting compound words according to the rules of part-of-speech word formation. Both approaches have their pros and cons.

The basic idea of extracting compound words based on statistical methods is: the higher the co-occurrence frequency of adjacent Chinese characters, the more likely it is an independent word. Therefore, the general process of this method is: (1) segment the electronic document according to a certain algorithm, and obtain each substring in it; (2) count the occurrence frequency of each substring or the probability of its left and right substrings appearing alone, etc. Judgment indicators; (3) According to whether these indicators reach the threshold to determine whether the substring is independent into a word. The advantage of this method is that it is not based on the dictionary, so it is not limited by the dictionary, the general recall rate is high, and new words can be extracted. The disadvantages are: (1) statistical methods are generally only suitable for automatic extraction of words in large corpora; (2) accuracy and recall cannot be guaranteed at the same time, and the threshold set for high accuracy will inevitably lead to lower recall ;(3) Grammar and lexical grammar are not considered when segmenting documents to obtain substrings, resulting in the final part of the substrings that are "non-words" are also mistakenly included in the prompt results, such as "system engineering", "knowledge management" wait.

The method based on part-of-speech analysis generally performs atomic word segmentation on the corpus based on the atomic word dictionary, and then takes the combination of atomic words (such as multiple nouns) as words according to the rules. Zhang Xin et al. proposed a method of automatically extracting terms based on part of speech to judge whether a Chinese character string is an independent word in the article "Research on Automatic Acquisition of Ontology Concepts Based on Rules and Statistics". The advantage of the word prompting method based on part-of-speech analysis is high accuracy; the disadvantage is that the recall rate is extremely low, which is limited by the accuracy and completeness of the rule set.

In order to overcome the defects of the above compound word extraction method and improve the performance of automatic word prompting, Yu Juan et al. proposed a method combining atomic word part-of-speech analysis and atomic word string in the article "word extraction method combined with part-of-speech analysis and string frequency statistics". The word prompt method of frequency statistics. The basic idea of this method is: the atomic word of a specific part of speech has a higher probability of participating in word formation, and the atomic word string with a higher co-occurrence frequency has a higher probability of "forming a word". Based on this idea, the method first processes electronic documents into a group of word strings composed of atomic words of a specific part of speech, then counts the frequency of these word strings and their substrings, and finally obtains the "word" words to achieve automatic extraction The purpose of the term in the document. However, this approach still has flaws. Although the recall rate of this method is satisfactory, there are a large number of "half words" in the result set, such as "management information" (existing in 'management information system'), "global" (existing in 'global enterprise'), etc. words, which affect the accuracy of the method.

Contents of the invention

In view of this, the purpose of the present invention is to provide a method for automatically extracting words from Chinese electronic documents, to solve the problem that the accuracy rate is affected by the existence of "half words" in the automatic extraction results, and to realize the computer to automatically and efficiently extract Chinese electronic documents. words in the document.

The present invention adopts following scheme to realize: a kind of method for extracting words automatically from Chinese electronic document, it is characterized in that comprising the following steps:

Step S01: processing the electronic document into a group of word strings composed of atomic words of a specific part of speech;

Step S02: Count the frequencies of these atomic word strings and their substrings, and use the atomic word strings that appear more than N times as candidate words, where N is a parameter that can be set;

Step S03: Delete the words that only appear as substrings in the set of candidate words, and obtain the set of words that appear in the document, so as to realize the purpose of automatically extracting words in the Chinese electronic document.

In an embodiment of the present invention, the implementation of step S01 includes the following steps:

S011: Perform atomic word segmentation and part-of-speech tagging on electronic documents, and obtain documents that have undergone atomic word segmentation and part-of-speech tagging;

S012: Delete useless atomic words to obtain a set of atomic word strings, including the following two steps to delete two kinds of useless atomic words:

S0121: Delete useless atomic words according to part of speech: replace the atomic words that do not participate in word formation with a first predetermined symbol, and in the output result, use a second predetermined symbol as an interval between atomic words, and use the above-mentioned atomic word strings the first predetermined symbol as a space;

S0122: Further delete the atomic words according to a list of inactive atomic words, and replace the inactive atomic words with the first predetermined symbol, thereby generating a new ordered set of atomic word strings.

In an embodiment of the present invention, the first predetermined symbol is a newline character, and the second predetermined symbol is a space.

In an embodiment of the present invention, the atomic word segmentation and part-of-speech tagging of the electronic document in the step S011 is completed by the Chinese Academy of Sciences word segmentation system ICTCLAS or the Harbin Institute of Technology word segmentation system IRLAS.

In an embodiment of the present invention, the step S02 is implemented using the following algorithm:

1) For each atomic word string AWS in the atomic word string set, perform step 2);

2) For each atomic word in the atomic word string, perform steps 3) and 4) sequentially;

3) Segment to obtain all substrings of AWS headed by the atomic word;

4) For each substring, execute step 5);

5) Determine whether the substring appears more than N times in the corpus, if yes, perform step 6); otherwise, perform step 7);

6) Remove the spacer in the substring to form a Chinese character string as a candidate word; at the same time save its frequency of occurrence;

7) Go back to step 2).

The invention designs and implements a method for automatically extracting words appearing in Chinese electronic documents. Compared with the existing prompting methods, this method: (1) When segmenting Chinese character strings, the atomic word is used as the step size, which avoids the wrong prompting caused by the segmentation of atomic words, such as "system worker", "Knowledge Management" and so on. (2) When extracting compound words, it shows high performance; compound words that are rarely used alone can also be extracted, such as "decision support" and so on. (3) Solved the problem of "half words" in the result set, and improved the accuracy of automatic word prompting. The effects and benefits of the present invention are: solving the practical problems and difficulties of low performance of automatic prompting and limited degree of automation. Efficient automatic word prompting method is the basis of automatic text processing and a powerful guarantee for applications such as information retrieval, text summarization, and content management. A good word extraction method can facilitate the automation and performance of the above work.

Description of drawings

Fig. 1 is a schematic flow chart of the method of the embodiment of the present invention.

Fig. 2 is a schematic flow chart of a specific method in another embodiment of the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

As shown in Figure 1, the present embodiment provides a method for automatically extracting words from a Chinese electronic document, which is characterized in that it includes the following steps: Step S01: processing the electronic document into a group of words composed of atomic words of a specific part of speech string; step S02: count the frequency of these atomic word strings and their substrings, and use the atomic word strings that occur more than N times as candidate words, where N is a parameter that can be set, and preferably this N can be 2; Step S03: Delete the words that only appear as substrings in the set of candidate words, and obtain the set of words that appear in the document, so as to realize the purpose of automatically extracting the words in the Chinese electronic document.

Specifically, referring to Fig. 2, the automatic word prompting method described in the present embodiment extracts the word collection in the Chinese electronic document in the following steps:

1. Perform atomic word segmentation and part-of-speech tagging on electronic documents to obtain documents that have undergone atomic word segmentation and part-of-speech tagging.

This step performs atomic word segmentation and part-of-speech tagging on the input electronic document. The word segmentation system ICTCLAS of the Chinese Academy of Sciences or the word segmentation system IRLAS of Harbin Institute of Technology can be used.

2. Useless atomic words are deleted to obtain a collection of atomic word strings.

Useless atomic words refer to those atomic words that generally do not participate in the formation of compound words. This step processes electronic documents that have undergone atomic word segmentation and part-of-speech tagging, and deletes two types of useless atomic words in two steps, and the output result is an ordered set of word strings composed of retained atomic words.

For the convenience of subsequent explanations, the following definitions are made:

Definition 1: Chinese atomic word string (AWS) is a finite sequence consisting of one or more Chinese atomic words. Recorded as AWS="aw _{1_} aw _{2_} ...aw _{n_} ", where aw _{1_} aw _{2_} ...aw _{n_} is the value of AWS, and aw _i (1≤i≤n) is an atomic word. The length of an atomic word string (denoted as AWSLen) refers to the number of atomic words constituting the atomic word string.

For example, "information_system_" is an atomic word string with a length of 2, which is formed after the atomic word segmentation of "information system".

Spaces can be used as separators between adjacent atomic words in the atomic word string. For clarity, an underscore "_" may be used to represent spaces.

Definition 2: A substring of an atomic word string is a subsequence of the atomic word string.

For example, "information_", "system_" and "information_system_" are substrings of the atomic word string "information_system_".

1) Delete according to part of speech. This step deletes useless atomic words according to part of speech. After inputting electronic documents that have undergone atomic word segmentation and part-of-speech tagging, this module retains those atomic words marked as specific parts of speech, and replaces atomic words that generally do not participate in word formation (such as prepositions, auxiliary words, etc.) with line breaks (or other predetermined symbols, which are not limited here), so that the output is an ordered set of atomic word strings, and the atomic word strings are composed of reserved atomic words. In the output results, a single space is used as an interval between atomic words, and a newline is used as an interval between atomic word strings.

2) Disable atomic word removal. This step further deletes the atomic words according to a list of inactive atomic words, and replaces the inactive atomic words with line breaks, thereby generating a new ordered set of atomic word strings. Stop using atomic words, that is, those words that may participate in the formation of compound words from the part of speech, but generally do not participate in the formation of words in actual situations, such as, is (verb), want (verb), provide (verb), many (adjective )wait.

3. The occurrence frequency of substrings is counted to obtain a set of candidate words.

In the previous step, the electronic document that has undergone atomic word segmentation and part-of-speech tagging is processed into an ordered set of atomic word strings. This step splits the substrings of these atomic word strings, and outputs the substrings that appear multiple times in the document as candidate words. These candidate words include atomic words, compound words, and some Chinese character strings that cannot be formed into words independently. The algorithm steps are as follows:

1) For each atomic word string AWS in the atomic word string set, execute 2).

2) For each atomic word in the atomic word string, execute 3) and 4) in sequence.

3) Segment to get all substrings of AWS headed by the atomic word.

4) For each substring, execute 5).

5) Determine whether the substring appears more than N times in the corpus (N is a parameter that can be set), if yes, go to 6); otherwise, go to 7).

6) Remove the spacers in the substring to form a Chinese character string as a candidate word; meanwhile save its frequency of occurrence. 7) Return to 2).

4. Delete the "half word" to get a set of words.

This step processes the set of candidate words, deletes the candidate words that only appear as substrings, and obtains the final automatic prompt result—the set of words in the electronic document. Candidate words that only appear as substrings refer to those substrings that appear in the document with the same frequency as their parent strings.

In the actual word extraction process, in order to improve the accuracy of the result, after the word set is automatically extracted, a manual correction step can also be added. Manual correction is the process of manually modifying the automatically extracted results by experts.

In order to allow those skilled in the art to better understand the present invention: take the document shown in Table 1 as an example.

Table I

After step 1, use the Chinese Academy of Sciences word segmentation system ICTCLAS to perform atomic word segmentation and part-of-speech tagging on the document. The document after segmentation is shown in Table 2.

Table II

Step 2 delete useless atomic words. The results are shown in Table 3:

Table three

Step 3 counts the occurrence frequency of substrings to obtain candidate words. The results are shown in Table 4.

serial number words Frequency of occurrence 1 enterprise 3 2 worldwide 3 3 global 2 4 global enterprise 2 5 communication 2 6 system 2 7 information 2 8 Information system 2 9 scope 2

Table four

Step 4 deletes the "half word" to obtain the word set for automatic word prompting by the method of the present invention. The results are shown in Table 5.

serial number words Frequency of occurrence 1 enterprise 3 2 worldwide 3 3 global enterprise 2 4 communication 2 5 Information system 2 6 scope 2

Table five

The above descriptions are only preferred embodiments of the present invention, and all equivalent changes and modifications made according to the scope of the patent application of the present invention shall fall within the scope of the present invention.

Claims (5)

1. method of therefrom extracting word in the message subdocument automatically is characterized in that may further comprise the steps:

Step S01: electronic document is treated to one group of word string of being made up of the atom word of specific part of speech;

Step S02: add up the frequency of those atom word word strings and substring thereof, occurrence number is surpassed N time atom word word string as candidate's word, but wherein N is setup parameter;

Step S03: only as the substring occurring words, obtain the set of occurring words in the document in the deletion candidate set of words, realize extracting automatically the purpose of the word in the Chinese electronic document.

2. extract the method for word in a kind of therefrom message subdocument according to claim 1 automatically, it is characterized in that: the implementation of described step S01 comprises the steps:

S011: electronic document is carried out atom word segmentation and part-of-speech tagging, obtain the document through atom word segmentation and part-of-speech tagging;

S012: delete useless atom word, obtain the set of atom word string, go on foot two kinds of useless atom words of deletion comprising following two:

S0121: delete useless atom word according to part of speech: the atom word that will not participate in organizing word replaces with one first predetermined symbol, among the output result, adopt one second predetermined symbol as at interval between the atom word, adopt described first predetermined symbol as the interval between the atom word string;

S0122: further delete the atom word according to an inactive atom word tabulation, the atom word of will stopping using replaces with described first predetermined symbol, generates the ordered set of new atom word string thus.

3. extract the method for word in a kind of therefrom message subdocument according to claim 2 automatically, it is characterized in that: described first predetermined symbol is newline, and described second predetermined symbol is the space.

4. extract the method for word in a kind of therefrom message subdocument according to claim 2 automatically, it is characterized in that: among the described step S011 electronic document is carried out atom word segmentation and part-of-speech tagging employing Words partition system ICTCLAS or Words partition system IRLAS and finish.

5. extract the method for word in a kind of therefrom message subdocument according to claim 1 automatically, it is characterized in that: described step S02 adopts following algorithm to realize:

1) for each the atom word string AWS in the atom word set of strings, execution in step 2);

2) for each atom word of atom word string, order execution in step 3), 4);

3) cutting obtains all substrings with the AWS headed by this atom word;

4) for each substring, execution in step 5);

5) judge whether the number of times that substring occurs surpasses N time in language material, if, execution in step 6); Otherwise, execution in step 7);

6) blank character of removing in the substring forms Chinese character string, as candidate's word; Preserve its frequency of occurrences simultaneously;

7) return step 2).

Images