JP2003303202A - Knowledge information collection system and knowledge information collection method - Google Patents

Abstract

(57) [Summary] [PROBLEMS] To efficiently collect various types of document files having different file formats from the Internet as knowledge information. According to a file format to be collected, which is set in a setting file, a Web collection module (111) provides a starting URL and a link destination to be followed.
It checks whether the file format of the document file to be collected matches the file format set in the setting file 203, and if it does not match, excludes the file from the collection target. Perform the process of collecting more.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a knowledge information collecting system and a knowledge information collecting method used in a knowledge management system, and particularly to a knowledge information collecting system and knowledge for collecting document information to be registered in a knowledge database from a network. Regarding information collection method.

[0002]

2. Description of the Related Art In recent years, the introduction of groupware for sharing information among a plurality of users has been promoted mainly in companies. E-mail systems and workflow systems are known as typical groupware,
Recently, a knowledge management system to support sharing of knowledge information is also being developed.

This knowledge management system is for accumulating and managing knowledge information such as personal know-how in a knowledge database. By combining it with a search function such as natural language search, the accumulated knowledge information can be efficiently stored. It is possible to make full use of it.

By the way, in such a knowledge management system, how to efficiently collect and store knowledge information is an important point. In particular, since Web information on the Internet is a treasure trove of knowledge, a mechanism for efficiently collecting necessary information from the Internet is needed.

[0005]

Recently, however, not only text-based document files such as HTML documents but also document files of various file formats created by various application programs have begun to be used for homepages. In this case, if the document file acquired from the Internet is directly registered in the knowledge database, it becomes difficult to search the document file in the knowledge database, and the knowledge cannot be effectively used.

The present invention has been made in consideration of such circumstances, and a knowledge information collecting system and a knowledge information collecting system capable of efficiently collecting various types of document files having different file formats from the Internet as knowledge information. The purpose is to provide a method for collecting knowledge information.

[0007]

In order to solve the above-mentioned problems, the present invention is a knowledge information collecting system for collecting document information to be registered in a knowledge database from a network, which should be registered in the knowledge database. A setting screen displaying unit for displaying a setting screen including a setting field of a starting point URL for collecting document information and a setting item of a collected file format that presents a plurality of types of collectable document file formats in a selectable manner; A setting file creating means for creating a setting file described based on the setting contents on the setting screen displayed by the setting screen displaying means, and the above described based on the description contents of the setting file created by the setting file creating means. An information collecting unit that collects document information to be registered in a knowledge database from the network, When acquiring the document information from the origin URL described in the setting file and the link destination to trace, the step checks whether the document information matches the collection file format described in the setting file, and sets the setting. It is characterized in that only the document information matching the collection file format described in the file is selected and collected from the network.

In this knowledge information collecting system, the information collecting means executes the collection of the document file from the network based on the setting information designating the collecting condition of the information from the network. When acquiring the document information from the origin URL described in the setting file and the link destination to be followed, the information collecting unit checks whether the document information matches the collection file format described in the setting file. , Only the document information matching the collection file format described in the setting file is selected and collected from the network. As a result, the file format of the document information to be collected can be selectively designated as needed, and the document information of the desired file format can be selectively designated and collected, and the knowledge information on the network can be collected efficiently. it can. Further, since desired knowledge information can be more appropriately narrowed down and collected, a higher-level knowledge database can be constructed easily, quickly and efficiently.

Further, according to the present invention, in a knowledge information collecting method for collecting document information to be registered in a knowledge database from a network, a plurality of kinds of document file formats that can be collected are set, and the document information to be collected is set. From a setting screen display step of displaying a setting screen having a setting item for setting a file format and a setting field of a starting point URL, a starting point URL set on the setting screen displayed in the setting screen displaying step and a link destination to be followed. For each piece of document information to be collected, a determination step of determining whether the file format of the document information matches the file format set on the setting screen, and the file format set in the determination step And excluding the document information that is determined not to be collected from the collection target. Only the document information that matches the format and wherein the collected from over the network.

According to this knowledge information collecting method, only the document information matching the file format set on the setting screen can be collected from the network, and the file format of the document information to be collected is required. It is possible to selectively collect the document information in any file format by selectively specifying it. As a result, desired knowledge information can be more appropriately narrowed down and collected, and a higher-level knowledge database can be constructed easily, quickly and efficiently.

[0011]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 shows the overall configuration of a knowledge management system using a knowledge information collecting system according to an embodiment of the present invention. This knowledge management system is for providing services such as collection, analysis, and search of knowledge information, and includes a Web collection system 11, a registration module 12, and a knowledge engine 13.
Etc. These Web collection system 1
1, the registration module 12, and the knowledge engine 13 are realized as a program executed on the server computer. The web collection system 11 and the registration module 12 constitute a knowledge information system for collecting knowledge information used in the knowledge management system. This knowledge information collecting system collects documents such as Web pages from the Internet / Intranet 30 and registers the contents of the documents in the knowledge database (knowledge DB) 131 of the knowledge management system. Web collection system 11
Collection module 111, management interface 112,
And a registration directory 113.

The Web collection module 111 is a program for collecting various document files on the Internet / Intranet 30 and outputting them in a format that can be registered in the knowledge DB 131. This Web collection module 111 has a multi-thread structure, and HTTP
Document file acquisition processing from the Internet / Intranet 30 by (HyperText Transfer Protocol) can be performed in parallel. HTTP is an information site on the Internet / Intranet 30 for each Web
From server 31 to HTML (Hyper Text Markup Languag
It is a communication protocol for acquiring the document described in e). The web collection module 111 uses the URL (Uniform Reload) specified by the setting information that specifies the web collection conditions.
Source Locator) document file is acquired from the Internet / Intranet 30, and if the acquired document file contains the URL of the link destination, the document file of that URL is further acquired. , The related document files are sequentially acquired from the Internet / Intranet 30. In this case, it is necessary to analyze the acquired document file in order to detect the link information (URL) to another document file. In the analysis process, not only the detection of the link information but also the knowledge Extraction of text data to be registered in the DB 131 is also executed.

The document file to be collected is created not only by a hypertext file (HTML file) capable of describing a URL that is link information to another file, but also by a text file (plane text) and various application programs. Document files in various file formats (for example, Portable Docu
ment Format files, document files created with word processor software, spreadsheet files, presentation data files created with presentation data creation software, etc.) Become.

The Web collection module 111 acquires attribute information to be registered in the knowledge DB 131 and extracts the above-mentioned text data for each document file collected from the Internet / Intranet 30. The attribute information acquired from each document file is, for example, the URL of the document file, file creation date / time (update date / time), and the like. Then, the Web collection module 111 creates a registration file in which the attribute information of each of the document files and the text data are described, and outputs it to the registration directory 113. Here, the registration file is
It means a file having a predetermined description format as a common interface for registering document information of each of a plurality of types of document files having different file formats in the knowledge DB 131. This registration file is used to register the document information of each of a plurality of types of document files in the knowledge DB 131 in a common format. In this embodiment,
XML (eXtensible Markup Langua) as a registration file
ge) is used.

The management interface 112 is a program for setting the contents of each Web collection process (hereinafter referred to as a Web collection process task) to be executed by the Web collection module 111, and the Web collection conditions for each Web collection process task. Of the Web collection module 111, and the functions of managing and presenting the collection status of each Web collection processing task. This management interface 112 allows the web server 22 to execute CGI (Common) so that an administrator user can perform necessary operations on the web browser 21 of the user terminal.
It is realized as an external program (CGI program) that can be started through the Gateway Interface.

Further, the Web collection system 11 includes a Web
As a file for managing and controlling the operation of the collection module 111, a lock file 201,
A control file 202, a setting file 203, a result file 204, and a log file 205 are provided.

The lock file 201 is a file for exclusive control for preventing double activation of the Web collection module 111. The control file 202 is a file used by the management interface 112 to stop the Web collection module 111. For example, in the case of interrupting a Web collection processing task that is being executed in response to an instruction from an administrator user. Used for. The administrator user can specify the interruption during the collection execution through the management interface 112.

The setting file 203 is a file for describing the setting information for designating the Web collection condition for each Web collection processing task, and here, a plurality of setting information corresponding to each of the plurality of Web collection processing tasks is set. Can be held. The web collection condition of each web collection processing task is set by the administrator user. In the Web collection condition, at least the origin address information (origin URL) indicating the location of the document file to be the origin of the information collection from the Internet / Intranet 30 and the collection range are specified only within the domain, or the URL for collection is specified. Includes multiple types of collection range information, such as specifying the pattern of URLs that are not collected, specifying the maximum number of document files or links that should be collected, and specifying the file format to be collected. Has been. This Web
We of the Web collection module 111 based on the collection condition
b The collection operation is controlled. That is, the Web collection module 111 sequentially collects the document file group related to the document file specified by the starting point URL within the range defined by the collection range information. The plurality of types of collection range information described in the setting file 203 and the setting means will be described later with reference to the GUI screen.

The Web collection condition also includes a knowledge database name that specifies the knowledge database of the registration destination. That is, in the knowledge DB 131, a plurality of knowledge databases having different types of accumulated documents are defined, and the plurality of knowledge databases are managed by respective knowledge database names. By designating the knowledge database name of the registration destination for each Web collection task by the setting information of the setting file 203 described above, the collected document information can be registered in any knowledge database in the knowledge DB 131.

Further, as the collection range information, a collection condition for recollecting can be set. Here, recollection means, for example, the same W as the Web collection module 111.
The eb collection processing is the second and subsequent Web collection processing when the eb collection processing task is periodically executed, and the Web collection processing when the once interrupted Web collection processing task is restarted.

The result file 204 is a file for managing a list of document files collected as a result of Web collection processing for each Web collection task, and here, a list of URLs acquired as a result of Web collection. ,
The number of acquired document files and the like are output for each Web collection processing task. This result file 204 is a web
It is used for the purpose of presenting the collection status of each collection processing task to the user and for the purpose of improving the efficiency of recollection. By using this result file 204, the Internet / Intranet 3 can be used among the document files collected in the past.
It is also possible to detect a document file deleted from the information site on 0 or a restart point when the Web collection process is interrupted midway.

The log fill 205 is a file for managing the success or failure of acquisition of a document file by the Web collection module 111 and the type of error.

The registration directory 113 is a knowledge DB 11
3 is a storage area for outputting the content of the document to be registered. The registration file 114 in which the attribute information and the text data extracted from each document file by the Web collection module 111 are described is output here.
The web collection module 111 has two processing modes, first and second, regarding extraction of text data.

In the first processing mode, analysis processing is performed on document files of all file formats, text data is extracted, and link information (URL) is detected (URL is detected only in HTML files. Target).
In the second processing mode, only the HTML file and the text file (PlainText) are analyzed to extract the text data and further the link information (URL) is detected (URL is detected only in the HTML file. ), For example, Portable Document F
The analysis process is not performed on a document file of another file format (hereinafter referred to as a content file) such as an ormat format file.

When the second processing mode is used, HTML
For PlainText and PlainText, the text data and attribute information are described in the registration file 114 and output to the registration directory 113. On the other hand, Portab
For files in other formats, such as files in the le Document Format format, the content file 115 is output to the registration directory 113 as it is, and the registration file 114 describes the attribute information of the file and the path name of the content file 115. It
Note that in the second processing mode, it is important that the text data extraction is also executed in the analysis process for the file that requires the analysis process for detecting the link information. The text data may be extracted only for the HTML file.

In both of the first and second processing modes, the Web collection module 111 executes the Web being executed.
When the processing of the collection task is completed, the registration module 12
A registration request file is issued to request registration of the document from the registration module 12. In the registration request file,
The file name of the registration file and the knowledge database name of the registration destination are described.

The registration module 12 registers the attribute information and text data of each document file collected by the Web collection module 111 in the knowledge DB 131.
That is, the registration module 12 acquires the corresponding registration file 114 in response to the registration request from the Web collection module 111, and extracts the attribute information and text data described for each document file on the registration file 114. , The registration destination knowledge database of the knowledge DB 131 is registered. The registration module 12 also has a text data extraction function, and the registration file 11
When the path name of the content file 115 is described in 4, the registration module 12 extracts text data from the content file 115 specified by the path name and registers it in the corresponding registration destination knowledge database of the knowledge DB 131. .

Further, the registration module 12 also executes a registration process of the documents collected by the collection modules other than the Web collection module 111. Other collection modules include, for example, a file collection module that collects document information from a file server of an electronic filing system, an RDB collection module that collects document information managed by the RDB database server, and an electronic bulletin board format community. There are a community collection module that collects the document information posted therein, a user collection module that the user uses to convert the document files of various application programs into the output format of the registration file (XML), and the like. The registration module 12 is commonly used by each of these collection modules. Registration module 1 for all collection modules
The above-mentioned XML-formatted registration file is used as an interface for 2.

The knowledge engine 13 performs a knowledge analysis process for utilizing the information accumulated in the knowledge DB 131. In this knowledge analysis process, a process of analyzing the features of each of a large amount of document information accumulated in the knowledge DB 131 to extract important words, a clustering process of classifying and organizing features into a plurality of categories, and the like are performed. This knowledge engine 13 includes a knowledge engine 132 for performing a natural language search.
Is provided for each user, and each user
By accessing the knowledge engine 132 from the browser 41 through the web server 42, the web browser 4
The knowledge accumulated in the knowledge DB 131 can be searched from above. Since the attribute information of each document stored in the knowledge DB 131 also includes the URL of the document, the original document is acquired from the corresponding information site and we
b It can be displayed on the browser 41. The search process can be performed by selecting a certain knowledge database name and targeting only the selected knowledge database, or by targeting all knowledge databases in the knowledge DB 131.

Next, the functional configuration of the Web collection module 111 will be described with reference to FIG. The web collection module 111 includes a collection control unit 301,
It has an attribute extraction unit 302, a text extraction unit 303, and a format conversion unit 304. Collection control unit 301
For each of all the setting information held in the setting file 203, the Web specified by the setting information
Run collection tasks one at a time. A group of related document files is sequentially collected from the Internet / Intranet 30 starting from the starting point URL designated by the setting information for each Web collecting task. The URL to the linked document included in the acquired document file is the URL list 30
5, the collection processing is recursively executed while acquiring URLs from the URL list 305. URL
The result file 204 described above can be used as the list 305. The collection range is limited by the collection conditions held in the setting file 203.

Each collected document file is first sent to the attribute extraction unit 302, where the attribute information of each document file is acquired. Next, the text extraction unit 303 analyzes the document and extracts the text data to be registered in the knowledge DB 131 and the link destination URL to be acquired next. For example, for an HTML file, extraction of text data is performed by extracting a portion other than the HTML tag. The extracted text is divided into a title and a body (body). To obtain the link destination URL, the A tag, the HREF value of the AREF tag, FRAME
This is performed by acquiring the SRC value of the tag, the IFRAME tag, the LAYER tag, and the REFRESH value of the META tag. When operating in the second processing mode described above, the processing of the text extraction unit 303 is performed only on the HTML file and the text file, and Port
It is not performed for a document file of another file format such as a file of the Document Format format.

The attribute information and text data obtained from each document file are sent to the format conversion unit 304, where they are shaped into an output format according to the description format of the XML registration file and output on the registration directory 113. In one registration file 114, for example, 1
Attribute information and text data for about 000 document files are described. The attribute information and path name of the document file for which text extraction has not been performed are described in the registration file 114.

Next, an example of the output format of the registration file 114 will be described with reference to FIGS. 3 and 4.

FIG. 3 shows an example of an output format when the text extraction unit 303 extracts text. Tag at the beginning of the file <? Xml version = "1.0" encod
ing = "Shift_JIS"?>, <KnowledgeSystem> indicates the beginning of the file, and the end tag </ KnowledgeSystem>
Indicates the end of the file.

Attribute information and text data of one document file is described in each record enclosed by <RECORD> and </ RECORD>. The meaning of the tag in each record is as follows.

MODE: Mode information for designating the operation mode of the registration module 12. This mode information instructs registration or deletion of document information (text data and attributes) in the knowledge DB 131 for each document file.
Either 2 = registration (overwrite) or 0 = delete. In case of deletion, only TYPE and UNIQUE of the following tags are output.

TYPE: Indicates the type of collection. In this example, it is always "Web collection". UNIQUE: A unique key for identifying the document file registered in the knowledge DB 131. Usually, the URL of the document is used as a unique key.

ORGDATE: Indicates the creation date (or update date) of the document file. TITLE: Indicates the title of the document file. The text data extracted from the title part of the HTML file becomes the title. No title is output for files other than HTML files. This title is output as the title of the document file on the search screen.

AUTHOR: Describe the host name (host address of URL) of the information site that owns the document file. DATE: Describe the date part of ORGDATE above. URL: URL of the document file. It has the same value as UNIQUE. BODY: The text data extracted from the document file is described.

FIG. 4 shows an example of an output format relating to a document file in which the text extraction unit 303 has not performed text extraction.

No text data is described in BODY.
In PATH1 in the area enclosed by BDYFILE></BDYFILE>, the path name to the content file 115 output on the registration directory 113 is described. DEL = 1 instructs the registration module 12 to delete the original file in the registration directory 113 after extracting the text data from the content file 115. We
When the b collection module 111 operates in the second processing mode, text data is described in BODY for HTML files and plain text, and BD for document files (content files) in other file formats.
The path name of the content file will be described in YFILE.

Next, referring to the flowchart of FIG.
Web collection module 1 for collected document files
A procedure of a series of processing executed in 11 will be described.

First, from the collected document files, the knowledge DB
Attribute information to be registered in 131 (URL, AUTHOR, ORGDATE,
DATE) is acquired (step S101). The attribute information can be acquired by using the value returned from the information site by HTTP, the value added in the collected document file, or the like. After that, in the second processing mode, the file type is determined based on the extension of the collected document file, and the file is an HTML file or a plain text file, or a file of other file format. Is determined (steps S102 and S103). The collected document file is H
If the file is a TML file or a plain text file (YES in step S103), the above-mentioned text extraction processing (text extraction and link URL detection in the case of an HTML file) is executed (step S10).
4) Then, the attribute information and the text data are described in the above-mentioned format on the registration file 114 (the text data is inserted into BODY) (step S105). On the other hand, if the file has a file format other than the HTML file or the plain text file (NO in step S103), the file is the registration directory 1
13 is output as it is (step S106), and then the attribute information and the path name of the file are registered file 1
The process described in 14 (the path name is described in BDYFILE) is performed (step S107).

If the document file of the target URL cannot be acquired from the Internet / Intranet 30, the content of the document file is already in the knowledge DB1.
Mode information (MOD
E) = 0 (delete) is described on the registration file 114.

Further, in the first processing mode, the processes of steps S102 and S103 are not performed, and the processes of steps S104 and S105 are executed for all the acquired files.

Next, referring to the flowchart of FIG.
The procedure of the registration process executed by the registration module 12 will be described.

The registration module 12, which has received the registration request from the Web collection module 111, retrieves the records one by one from the registration file (XML file) specified by the registration request file from the Web collection module 111, and Process. First, is the mode information in the processing target record MODE = 0 or MODE = 2
Is checked (steps S111 and S11).
2).

When MODE = 2, the registration module 12
Is the data item (TYPE, UNIQUE, ORGDATE, TITLE, AUTH) described in each tag according to each tag in the record.
The contents of OR, DATE, URL, and BODY) are registered in the registration destination knowledge database in the knowledge DB 131 specified by the registration request file (step S113). If the text data does not exist in the BODY tag, the text data will not be registered. Then, it is judged whether or not the path name is described in the tag of BDYFILE (step S114). If the path name is described (YES in step S114), the corresponding content file is acquired from the storage area specified by the path name (step S115), and the process of extracting text data from the content file is executed. Performed (step S1
16). Then, the extracted contents are registered in the registration destination knowledge database as text data of the corresponding document file (step S117).

When MODE = 0, the attribute information and text data of the registered document file designated by UNIQUE in the record are searched from the knowledge DB 131, and the registered content is deleted from the registration destination knowledge database (step S118).

FIG. 7 shows the relationship between the state of the document file (Web content) on the Internet / Intranet 30 and the registration / deletion process to be performed on the knowledge DB 131. In the present embodiment, the following processing is performed based on the policy of reflecting the state of the latest Web content on the Internet / Intranet 30 in the knowledge DB 131 as much as possible.

(1) The updated Web content is overwritten on the knowledge DB 131 (MODE = 2). Updated W
When registering the attribute information and text data of the eb content, the UNIQUE is the same as the UNIQUE of the pre-update Web content already registered in the knowledge DB 131. Therefore, MODE = 2 updates (overwrites) the attribute information and text data of the pre-update Web content already registered in the knowledge DB 131 with the updated Web content attribute information and text data.

(2) The added Web contents are additionally registered in the knowledge DB 131 (MODE = 2). When registering the attribute information and text data of the added Web content, the UNIQUE is not registered in the knowledge DB 131. Therefore, by setting MODE = 2, the attribute information and text data of the added Web contents are stored in the knowledge DB 13.
1 can be additionally registered.

(3) The deleted Web contents are also deleted from the knowledge DB 131 (MODE = 0).

Next, the functions provided by the management interface 112 will be specifically described. As described above, the management interface 112 sets the Web collection task settings /
It is a user interface for managing and executing,
1) A function of creating a plurality of settings of a web collection task and saving them in the setting file 204. 2) A list of collection settings,
The administrator user is provided with a function of performing management work such as deletion, 3) a function of starting and ending (suspending and resuming) collection, and 4) a function of confirming the status and achievement of collection in real time.

The screen displayed on the Web browser 21 of the administrator user by the management interface 112 is composed of three screens of "list screen", "setting screen" and "status screen" as shown in FIG. When the management interface 112 is activated, the list screen is first displayed.
The three screens transit according to the operation performed by the administrator user on each screen. The list screen displays a list of currently created settings and the latest collection result of each setting. On this list screen, for example, when a created setting is selected and the selected setting transitions to the setting screen or status screen related to the setting, addition / deletion of the setting content can be performed on the setting screen. It is possible to perform, and on the status screen, it is possible to specify activation and termination (suspension and resumption) of the setting. If a new setting is selected on the list screen, the setting screen is displayed, and a process for creating a new setting can be performed on the setting screen. The list screen, the setting screen, and the status screen will be specifically described below.

FIG. 9 shows an example of the list screen.
On this list screen, as shown in the figure, "List" anchor 5
01, “Setting anchor” 502, and “Status” anchor 503 are displayed. These “list” anchor 501, “setting anchor” 502, and “status” anchor 503 are links to the list screen, setting screen, and status screen, respectively. When the “list” anchor 501 is clicked on the list screen, the display content of the list screen is updated to the latest state. Further, on the list screen, the "new setting" field 50
4, "Create" button 505, and status list display section 50
6 is provided. Enter the setting name of the Web collection task to be newly created in the "New setting" field 504,
When the "Create" button 505 is clicked, the screen changes to the setting screen and new setting information regarding the setting name can be created.

The status list display unit 506 displays a list of collection statuses for each of the Web collection tasks that have already been created and registered, based on the contents held in the result file 204 described above. The following items are displayed for each Web collection task.

"Setting name": Name of registered setting "Knowledge database name": Name of database to register collected data "Status": Current status of setting (uncollected / collecting / finished / interrupted / abnormal) Either end) "Start": Date and time of last start "End": Date and time of last end "File": Number of files collected by the previous task execution
(This time) and the maximum value of the total number of files collected so far (whole) As described above, in this embodiment, a plurality of Web collection tasks can be set and Multiple We
b The collection status of each collection task can be confirmed on the list screen.

Next, an example of the setting screen will be described with reference to FIG. On the setting screen, as shown in the figure, in addition to the “list” anchor 501, the “setting anchor” 502, the “status” anchor 503, a “setting name” field 601,
“Knowledge database name” field 602, “proxy server” setting form 603, “collection file format” setting form 604, “acquisition request” setting form 605,
An “authentication information” setting form 606 and a “collection site” setting form 607 that constitutes a collection site setting screen are provided, and the following setting editing operations can be performed using these.

(1) Setting name: "Setting name" field 6
In 01, the setting name designated by creating a new setting or selecting an existing setting on the list screen is displayed. (2) DB selection: By inputting the knowledge database name in the “knowledge database name” field 602, the registration destination knowledge database for registering the collected data is selected.

(3) Proxy: The proxy server used for collection is set using the "proxy server" setting form 603. (4) File format to collect: “Collecting file format”
A file format to be collected is selected on the setting form 604. In this embodiment, the HTML and TEXT files are selected by default (HTML and T
(You cannot cancel the specification of the EXT file.) In addition to this, for example, Portable Document Format
Attachment files of various general-purpose file formats, such as t format files, document files created with word processor software, files created with spreadsheet software, presentation data files created with presentation data creation software, etc. You can choose. In the present embodiment, an example in which the file formats of the collectable document files are set to 6 types is shown, and each can be arbitrarily designated by a check box. For example, 6 file formats (HTM
By specifying all (L, PDF, PPT, DOC, XLOS, JBW) with check boxes, it is possible to collect document files of all the specified file formats. Further, by deselecting the check box for the unnecessary document file in the file format, the document file in the file format is deleted from the knowledge DB 131.

(5) Acquisition request: The timeout time and the number of retries for URL acquisition are specified on the "acquisition request" setting form 605. (6) Authentication Information: On the “authentication information” setting form 606, the authentication information required when accessing a URL that requires user authentication is set.

(7) Collection site: The following items are set on the "collection site" setting form 607.

Setting of starting URL: The URL which is the starting point of collection is specified. In this embodiment, a maximum of 100 excluding half-width characters and special characters is set on the setting field of the origin URL.
You can enter up to 0 characters.

Setting the upper limit of the number of link stages: Designate the maximum number of link stages from the collection origin URL. -Upper limit setting of the number of files: Specify the maximum number of files to be collected. -Collecting range: If "Collect only in domain" is specified, only the domain of the origin URL is collected. The default is no specification. This "collect only within the domain"
Is specified in the check box, the document information is collected from the corresponding information site on the Internet / Intranet 30 for the origin URL set in the origin URL setting field only within the domain.

URL pattern to be collected: Designate the character string pattern of the URL to be collected. Of the URLs to be collected, only the files of URLs that include the specified character string pattern are collected. In the present embodiment, it is possible to set a maximum of 1000 bytes, one pattern per line, in the setting field for setting the pattern of the collected URL.

URL pattern not to be collected: Specifies the pattern of the URL character string that should be excluded from the collection target. Among the URLs to be collected, the URL file including the specified character string pattern is not collected. In the present embodiment, one pattern per line is set in the setting field for setting the pattern of URLs that are not collected.
A maximum of 1000 bytes can be set.

If there is a URL that matches both the collected pattern and the uncollected pattern, the uncollected pattern takes precedence. That is, the pattern to collect,
If there is a URL that matches both of the patterns that are not collected, the file of that URL is not collected.

Recollection settings: Specify the behavior when the collection task to be set recollects from the same origin URL. There are three modes for recollection ("collect all files newly", "collect only uncollected files", "collect only updated files (with delete function)",
Any one mode can be specified. here,
The "collect all files newly" mode means a mode in which all the collection target URLs are collected again. That is, regardless of whether it has been acquired in the past, it is newly collected from the origin URL. Also, "collect only uncollected files"
The mode is a mode in which previously collected URLs are not collected. The "collect only updated files (with deletion function)" mode is a mode in which only the document files that have been updated from the previous collection point are collected. The document whose update is to be checked is the URL collected in the previous collection more than the specified number of days before the current collection. "Update check frequency"
The value entered in the field becomes the specified number of days. The default number of designated days is "1 day". In the previous collection,
If the URL collected more than the specified number of days before the present is deleted from the corresponding information site on the Internet / Intranet 30 and the document file cannot be acquired from the URL, the registration information corresponding to the document file Is deleted from the knowledge DB 131.

FIG. 11 shows an example of the status screen. For example, selecting the “status” anchor 503 on the list screen or the setting screen causes a transition to the status screen of the currently displayed task. The "setting name", "knowledge database name", "state", etc. of the task are displayed on this status screen. Next to the “setting name”, a button 701 for instructing start / stop (restart / interrupt) of collection is displayed. In "Status", the status of the task (collecting / finished / not collected / suspended / abnormally terminated) is displayed, and further, the starting URL of the task, the number of link stages collected (current / maximum / upper limit), and the number of files The status information 702 such as (current / total / upper limit) is also displayed.

When the status of the task is not collected, finished, or suspended, the button 701 displays "collection started", as shown in FIG. "Start collection" button 70
When 1 is selected, the execution of the task is started by the web collection module 111. If the current state is ended or suspended, recollection will be started. If the current status of the task is being collected,
As shown in FIG. 12, the button 701 displays “collection interruption”. If you select the “Collect collection” button 701,
Collection tasks that are running are suspended. The list of URLs collected before the interruption is held in the results file 204.

Next, with reference to FIG. 13 to FIG. 15, the interruption and resumption (recollection) operation of the collection process by the button operation on the status screen will be specifically described.

The suspend / resume function is a function used when the Web collection processing in progress is temporarily stopped and restarted at a later date. W by Web collection module 111
The eb collection process is performed by the internet / intranet 30
Each time a document file is acquired from the URL, the URL, a status code indicating success or failure, and the like are written in the result file 204 as the progress status information. If the acquired document file has a URL of the link destination, the URL is added to the result file 204, and the process is repeated until there is no uncollected URL in the result file 204. If the "collection suspension" button 701 is pressed during this collection processing, the Web collection task in execution is suspended at that point. Then, later, when the "collection start" button 701 is pressed with the interrupted Web collection task selected, the interrupted Web collection task is restarted and recollection is started. For example, when the “collect only uncollected files” mode is selected as the recollection mode, the uncollected URs held in the result file 204 are collected.
The collection process will be restarted from L.

FIG. 14 shows an example of a URL list held as progress status information in the result file 204. As shown in the figure, in the result file 204, in addition to the “URL” of each URL to be acquired, “collection date”, “collection date and time”, “status code”, “starting URL”, and “number of link stages” are included. Will be recorded. "Number of links" is
Indicates the number of links in the acquisition target URL when viewed from the starting URL. For the uncollected URLs, the information of “collection date”, “collection date and time”, and “status code” is blank.
This makes it possible to detect the restart point when the Web collection process is interrupted midway. The "status code" indicates the success or failure of the collection process, which is an HTTP status code returned from the partner web server from which the web page is acquired. The type of this HTTP status code is "200" indicating that the file acquisition was successful (normal end), "403" indicating an error due to the access authority, and the corresponding file cannot be found (for the request to acquire the deleted document file). “Response”) includes an error “404”, a timeout error “408”, and the like.

The flowchart of FIG. 15 shows a series of processing procedures from the start of Web collection to the interruption thereof.

The Web collection module 111 first
The document file specified by the origin URL is acquired (step S121), and the result (collection date, collection date and time, status code, origin URL, link stage number, URL (same as origin URL) is registered in the result file 204. The progress status information is updated (step S122) If the acquisition is successful, the knowledge DB 13 is acquired from the collected document files.
The attribute information to be registered in 1 is acquired (step S12
2) Next, text extraction processing (text extraction and detection of link destination URL in the case of HTML file) is executed (step S124). When the link destination URL is detected, the URL, the starting point URL, and the link stage number described above are additionally registered in the URL list of the result file 204 (step S125). After that, a process of describing the attribute information and the text data on the registration file is performed (step S126). Here, it is determined whether or not there is an interruption instruction (step S127). If there is no interruption instruction (NO in step S127), it is determined whether or not the uncollected URL is registered in the result file 204.
It is determined whether the collection process is completed (step S
128).

If uncollected URLs are registered in the result file 204 (NO in step S128), the processes from step S121 described above are repeated. That is, the document file specified by the URL added to the result file 204 is acquired (step S121),
The result (collection date, collection date and time, status code) is registered in the result file 204 to update the progress status information (step S122). If the acquisition is successful, the attribute information to be registered in the knowledge DB 131 is acquired from the collected document file (step S122), and then the text extraction process (text extraction and detection of link destination URL in the case of HTML file) is executed. (Step S12
4). If the linked URL is detected, its UR
L, the starting URL and the number of links are the result file 20
4 is additionally registered in the URL list (step S12).
5). After that, a process of writing the attribute information and the text data on the registration file 114 is performed (step S12).
6). Such Web collection processing is performed by the setting file 20.
It is recursively executed until the collection is completed within the collection range of the Web collection condition set to 3. When the "collection suspension" button 701 is pressed during this Web collection, the management interface 112 is accessed through the control file 202.
Issues an interruption instruction to the Web collection module 111 (YES in step S127), and the Web collection task being executed by the Web collection module 111 is interrupted.

Here, as the collection range, according to the file format of the collection target set on the “collection file format” setting form 604 of the setting screen shown in FIG. 10, regarding the Web collection processing executed by the Web collection module 111, This will be described with reference to the flowchart shown in FIG.

When "Setting anchor" 502 is clicked on the list screen shown in FIG. 9, the list screen shown in FIG. 9 is changed to the setting screen shown in FIG. 10 as shown in the screen transition diagram of FIG. By setting the “collection file format” setting form 604, the “collection site” setting form 607, etc. on the setting screen, the setting file 203 including the setting contents of each setting form is created (step S241, 242).

In the present embodiment, the "collected file format" setting form 604 sets the file format of the document information (document file) to be collected. Here, 6 types of document file formats that can be collected (HTML, P
DF, PPT, DOC, XLOS, JBW)
When there is a document file format you want to collect other than the HTML and TEXT files set by default, by specifying the document file format you want to collect with the check box, you can collect the document files of all the specified file formats. Can be

Further, in the "collection site" setting form 607 on the above setting screen, the starting point URL of the site to be collected is set in the starting point URL setting field. At this time, at the same time as the setting of the starting URL, the setting of the collected URL pattern, the setting of the non-collected URL pattern, the setting of the collection only within the domain, the upper limit setting of the number of document files to be collected, the upper limit setting of the link stage It is possible to set various collection ranges such as, etc., and by combining these settings with the setting of the above document file format, in addition to the document file format to be collected, URL patterns to be collected, URL patterns not to be collected, only patterns within the domain, Each narrowing (filtering) function such as the upper limit of the number of document files and the upper limit of the number of link stages is possible, but in the present embodiment, the setting of the collection range other than the setting of the file format to be collected is not particularly mentioned.

After setting the "collection file format" setting form 604 and the "collection site" setting form 607 as described above on the setting screen shown in FIG. 10, "status anchor" is displayed on the setting screen. When 503 is clicked, as shown in the screen transition diagram of FIG. 8, the setting screen shown in FIG. 10 transits to the status screen shown in FIG. 11 (step S243), and the “collection start” button is pressed on the status screen. If 701 is selected, the web shown in FIG.
The collection process is started, and in the Web collection process,
Web collection can be performed by narrowing the collection range according to the setting contents of the “collection file format” setting form 604.

In order to narrow down the collection range, in this embodiment, according to the file format set in the "collected file format" setting form 604, the document information (document file) to be collected is selected and the collection processing is performed.

In this collection processing, the starting point URL is read by referring to the setting file 203 (step S24).
4 to S246), the site specified by the URL, that is, the Web
The server 31 is accessed to make a file request. In response to this request, the web server 31 returns the content type of the collection target file.

The Web collection module 111 uses the above URL
When the content type is acquired from each Web server 31 designated in step S247, the contents of the content type are checked (step S247), and the document file to be collected is the file set in the "collected file format" setting form 604. It is determined whether or not the format matches (step S248).

Here, if the document file to be collected matches the file format set by the "collected file format" setting form 604 (step S2).
If YES at 48, the collecting process is started (step S2).
49).

The document files to be collected are
If it does not match the file format set in the "collected file format" setting form 604 (step S24)
8), the link destination is checked, and if there is a link destination (NO in step S250), each Web of the link destination
The same access as described above is made to the server 31 (steps S246, S247, ...).

In this way, the Web collection module 1
Reference numeral 11 selects document files to be collected according to the file format to be collected set in the setting file 203, and collects document information from the network. As a result, a document file of a required file format can be designated and collected on the network, and by narrowing down the document files to be collected, knowledge information on the network can be efficiently collected.

Further, in addition to the above-mentioned specification of the file format to be collected, the setting of the pattern of the URL to be collected, the setting of the pattern of the URL not to be collected, the setting of the collection only within the domain, the number of document files to be collected Upper limit setting,
Multiple types of collection range can be set at the same time, such as setting the upper limit of the number of link stages and setting of the file format to collect.By filtering the document files to be collected by combining these, you can obtain the desired knowledge information more appropriately. Therefore, it is possible to construct a higher-level knowledge database easily, quickly and efficiently. For example, the above-mentioned file format to be collected and U to be collected
By combining and setting the RL pattern and the URL pattern that is not collected, the URL pattern that is collected and the UR that is not collected are in the specified file format.
Desired knowledge information can be collected by designating the L pattern. Further, by setting the file format to be collected and the collection only in the domain, the knowledge information in the specified file format can be collected only in the domain of the origin URL. Furthermore, in addition to this, by setting the upper limit value of the number of document files, the upper limit value of the number of link stages, etc., the number of link stages, the number of files, etc. to be collected can be arbitrarily regulated.

Since all the functions of the knowledge information collecting system of this embodiment are realized by a computer program, the computer program is stored in a computer-readable storage medium, and this computer program is stored in the storage medium. It is possible to easily obtain the same effect as that of the present embodiment only by installing the program in a normal computer that can be connected to a computer network and executing the program.

Further, the present invention is not limited to the above-mentioned embodiment, and can be variously modified at the stage of implementation without departing from the spirit of the invention. Furthermore, the embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some constituent elements are deleted from all the constituent elements shown in the embodiment, the problem described in the section of the problem to be solved by the invention can be solved, and the effect described in the section of the effect of the invention can be solved. When the above is obtained, the configuration in which this constituent element is deleted can be extracted as the invention.

[0092]

As described above, according to the present invention,
Various types of document files having different file formats can be efficiently collected as knowledge information from the Internet.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of a knowledge information collecting system according to an embodiment of the present invention.

FIG. 2 is an exemplary block diagram showing the functional configuration of a Web collection module provided in the knowledge information collection system of the same embodiment.

FIG. 3 is a diagram showing an example of an output format of a registration file used in the knowledge information collecting system of the embodiment.

FIG. 4 is an exemplary view showing another example of an output format of a registration file used in the knowledge information collecting system of the embodiment.

FIG. 5 is an exemplary flowchart showing a processing procedure of a Web collection module provided in the knowledge information collection system of the embodiment.

FIG. 6 is an exemplary flowchart showing a processing procedure of a registration module provided in the knowledge information collecting system of the embodiment.

FIG. 7: We in the knowledge information collecting system of the same embodiment
The figure which shows the relationship between b content and its registration processing.

FIG. 8 is a diagram for explaining three screens provided by the knowledge information collecting system of the embodiment and their transitions.

FIG. 9 is an exemplary view showing an example of a list screen used in the knowledge information collecting system of the embodiment.

FIG. 10 is an exemplary view showing an example of a setting screen used in the knowledge information collecting system of the embodiment.

FIG. 11 is an exemplary view showing an example of a situation screen used in the knowledge information collecting system of the embodiment.

FIG. 12 is an exemplary view showing an example of a situation screen used in the knowledge information collecting system of the embodiment.

FIG. 13 is an exemplary view for explaining a suspending / resuming operation of the Web collecting process in the knowledge information collecting system of the embodiment.

FIG. 14 is a diagram showing an example of contents of a result file provided in the knowledge information collecting system of the embodiment.

FIG. 15 is an exemplary flowchart showing a processing procedure from the start to the interruption of the Web collection processing in the knowledge information collection system of the embodiment.

FIG. 16 is an exemplary flowchart showing a procedure of setting a file format to be collected and a Web collecting process associated with the setting in the knowledge information collecting system of the embodiment.

[Explanation of symbols]

11 ... Web collection system 12 ... Registration module 13 ... Knowledge engine 30 ... Internet / Intranet 111 ... Web collection module 112 ... Management interface 113 ... Registration directory 131 ... Knowledge database 132 ... Search engine 201 ... Lock file 202 ... control file 203 ... Setting file 204 ... Result file 205 ... log file 301 ... Collection control unit 302 ... Attribute extraction unit 303 ... Text extraction unit 304 ... Format conversion unit 604 ... “Collection file format” setting form 607 ... "Collection site" setting form

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Koji Shioda             2-9 Suehiro-cho, Ome City, Tokyo Stock Market             Company Toshiba Ome Factory F-term (reference) 5B075 ND03 NK37 NK44

Claims (8)

[Claims] 1. A knowledge information collecting system for collecting document information to be registered in a knowledge database from a network on the basis of information set by the setting means. A knowledge information collecting system including a setting means, and a collecting processing means for selecting and collecting only document information that matches the file format set by the setting means. 2. In a knowledge information collection system for collecting document information to be registered in a knowledge database from a network, a plurality of types of document file formats that can be collected are set, and a file format of the collected document information is set. A setting item to be set, and a graphic user interface displaying a setting screen having a setting field of a starting point URL, and a starting point URL set on the setting screen displayed by the graphic user interface and document information collected from a link destination to be traced A collection filter that checks whether or not the file format of the document information matches the file format set by the graphic user interface, and excludes the document information that does not match the set file format from the collection target. On the setting screen A knowledge information collecting system, characterized in that only the document information that matches the file format set in 1. is collected from the network. 3. In a knowledge information collecting system for collecting document information to be registered in a knowledge database from a network, a starting URL setting field for collecting document information to be registered in the knowledge database, and a collectable A setting screen display means for displaying a setting screen including the setting items of the collection file format that presents a plurality of different document file formats in a selectable manner, and the setting contents on the setting screen displayed by the setting screen display means. Setting file creating means for creating a setting file described in, and information for collecting document information to be registered in the knowledge database from the network based on the description content of the setting file created by the setting file creating means. And a collecting unit, wherein the information collecting unit is provided with a starting point URL described in the setting file. When the document information is acquired from the traced link destination, it is checked whether the document information matches the collection file format described in the setting file, and the document matching the collection file format described in the setting file. A knowledge information collecting system characterized in that only information is selected and collected from the network. 4. The setting items of the collection file format include:
The knowledge information collecting system according to claim 3, wherein a file format that can be registered in the knowledge database is initialized. 5. On the setting item of the collection file format,
When a file other than the initially set file format is unset, processing means for deleting document information of the unset file format registered in the knowledge database from the knowledge database is provided. The knowledge information collecting system according to claim 4, characterized in. 6. The setting screen displayed by the setting screen display means includes setting items whose collection range is within the domain,
Includes at least one of items to set the character string pattern of URLs to be collected, items to set the character string pattern of URLs to not be collected, items to set the upper limit of the links to be collected, and items to set the upper limit of the number of files to be collected. 4. The knowledge information collecting system according to claim 3, wherein the setting content of each item is reflected in the setting content of the setting item of the collection file format. 7. A knowledge information collecting method for collecting document information to be registered in a knowledge database from a network, wherein a plurality of types of document file formats that can be collected are set, and a file format of the collected document information is set. A setting screen display step for displaying a setting screen having a setting item for setting and a setting field for the starting point URL, and the starting point URL set on the setting screen displayed in the setting screen displaying step and the document information collected from the link destination to be followed. For each, the file format of the document information is
A determination step of determining whether or not the file format set on the setting screen matches; a step of excluding document information determined not to match the set file format from the collection target from the determination step; A knowledge information collecting method, characterized in that only the document information that matches the set file format is collected from the network. 8. A knowledge information collecting method for collecting document information to be registered in a knowledge database from a network, wherein a setting field of a starting point URL for collecting document information to be registered in the knowledge database and a collectable The setting screen display step for displaying the setting screen including the setting items of the collection file format that presents multiple types of document file formats that can be selected, and the setting contents on the setting screen displayed by the setting screen display step. And a setting file creating step of creating a setting file described in 1., and collecting document information to be registered in the knowledge database from the network based on the description content of the setting file created in the setting file creating step. And in the collecting step, When acquiring the document information from the point URL and the link destination to be followed, it is checked whether the document information matches the collection file format described in the setting file, and it matches the collection file format described in the setting file. A method for collecting knowledge information, characterized in that only the document information to be collected is selected and collected from the network.