CN102236654A - Web Invalid Link Filtering Method Based on Content Correlation - Google Patents

Abstract

The present invention discloses a method for filtering invalid Web links based on content relevance. The method firstly utilizes text position information in a web page to remove irrelevant advertising links and navigation links in the web page by statistical methods; then, the content of the web page and the content of the web page to which the link points are analyzed for relevance, and invalid links with irrelevant content are removed. The invalid link filtering method can effectively remove invalid links, and PageRank calculation is performed on the purified link structure diagram, and the page ranking result can be greatly improved, including the improvement of the quality of the top-ranked web pages, and the introduction of more high-value websites.

Description

Web Invalid Link Filtering Method Based on Content Correlation

technical field

The invention relates to a method for filtering useless links in Web pages, in particular to a method for filtering useless links in Web pages based on content correlation analysis, and belongs to the technical field of Internet search.

Background technique

With the rapid development of the Internet, search engines for Internet information queries are playing an increasingly important role. For a search engine, its main task is to find relevant web pages and return them to users in order of page importance. With the growth of the number of Web pages, the richness of page content and the variety of page links, search engines began to become more and more "powerless". There are many reasons for this, the most important of which is the increasing proliferation of invalid links in Web pages.

After analysis, links in web pages can be divided into the following four categories:

Artificially generated links: Most of these links are manually created by comparing the content of two web pages based on their relevance, and are classified as "related links" by the web page creators, so most of these links have Strong recommendation. There is also a part of the content of the web pages pointed to that is not related to the content topic of this web page, but is only slightly related to a certain point.

Navigation link: This type of link is generated by the webpage creator using the corresponding module, which is basically the same for webpages under the same site, mainly to allow users to access between different areas of this website. These links play a certain navigation role for user visits, but have nothing to do with web page relevance recommendations.

Advertising links: This type of link is generated according to some dynamic functions in the webpage. It is generally added for the commercial benefit of the website and accounts for a large proportion of the links. Especially for com-type webpages, this part of the link Accounted for more than half. These links basically do not contribute to recommending content-related web pages.

Favored links: This type of link mainly refers to the link between the main webpage and the sub-webpage it points to belong to the same site. It is for the website creator to recommend some new or more concerned webpages in this website and increase their click-through rate. And added to the web page.

Fig. 1 is a news class webpage intercepted on Sina, wherein the link in ⑤ is an artificially generated link (the above-mentioned first category). The links in ① are links with navigation significance in the website (the second category above), and they point to the homepages of other websites of Sina respectively; the links in ② and ⑥ are partial links (the third category above), It is generated to recommend the latest news of the day. It can be seen from the content that they point to some web pages that have nothing to do with this web page; the links in ④ and ⑦ are advertising links added by the website for economic benefits (the above-mentioned No. four categories).

After analysis, the inventor believes that the types of links that are not recommended, including links in the first category that do not have topic relevance, links in the second, third, and fourth categories, are collectively referred to as "invalid links"; Links within a class that have topical relevance are called "active links".

Link analysis is one of the most successful methods for ranking Web pages, including Google, Yahoo! Many search engines, including Google, use this method, combined with factors such as anchor text and word frequency statistics, and achieved great success. The success of the link analysis method largely depends on the validity of the web page links, or on the rationality of the following assumptions: when there is a link from web page A to web page B, it means that the author of web page A thinks that web page B is Content is important, and generally speaking, the content of pages A and B have related topics. It can be said that this assumption of content relevance is the basis on which the link analysis method lives.

In the early days of Internet development, the links in web pages basically conformed to the assumption of content relevance, and the correlation transfer between web pages was meaningful. However, with the continuous development of web technology and the continuous expansion of the number of web pages, more and more web pages are automatically generated by web page generation tools, so many links lose their relevance meaning, resulting in an increasing proportion of invalid links. At the same time, with the application of search engines, many website managers have introduced a large number of useless links in order to improve the ranking in search engines, and many spam sites have appeared. On the other hand, most of the current commercial websites take commercial interests as the ultimate goal, which leads to the introduction of a large number of advertising links. Based on the above reasons, the content relevance and recommendation significance of links in the Web have been seriously threatened. If it is not processed, the constructed link structure graph cannot correctly reflect the association relationship between web pages, and the sorting results obtained based on such a link graph will no longer be true and valid.

Contents of the invention

The technical problem to be solved by the present invention is to provide a web invalid link filtering method based on content correlation. The link filtering method improves the effect of invalid link filtering by constructing a more reasonable link structure graph and then performing link analysis.

In order to realize above-mentioned purpose of the invention, the present invention adopts following technical scheme:

A kind of Web invalid link filtering method based on content correlation is characterized in that comprising the steps:

(1) Use the text position information in the webpage to remove irrelevant advertising links and navigation links in the webpage through statistical methods;

(2) Carry out correlation analysis on the content of the web page and the content of the web page pointed to by the link, and remove invalid links with irrelevant content.

Wherein, in the step (1), the HTML document is first converted into a DOM tree structure, and then the smallest subtree containing the main content and links related to the topic is searched in the DOM tree structure to obtain the required link information.

For the DOM tree structure, first utilize the block node to divide the DOM tree into each subtree, calculate the link ratio in each subtree, and compare it with a predetermined threshold; if it is less than the threshold, then the block is set as the main body block, Then backtrack to find the nearest parent block node that contains the block, take the parent block node as the target node, and output the link in the parent block node as the basis for subsequent analysis.

In the step (2), before performing the content correlation analysis of the webpage, the text of the webpage is preprocessed, and the content representing each text is extracted for comparison.

The process of text preprocessing includes the following steps:

First perform text segmentation, then count the frequency of words in the text, calculate the TF-IDF vector, and form a vector space model corresponding to the text set; use the feature vector of the text to calculate the content similarity between each text, and use the content similarity degree to remove irrelevant links in web pages.

The content similarity is determined by the degree of overlap of terms contained in the feature vectors of each text.

Alternatively, the content similarity is determined by the cosine of the included angle in the feature vectors of each text.

In described step (2), described content correlation analysis comprises three layers of operations: the first layer is to carry out content correlation analysis according to entry text; The second layer is to carry out content correlation analysis according to the title of webpage; The third layer is Carry out content correlation analysis according to the content of the webpage text; if it is concluded that the theme of the webpage is irrelevant in these three layers, delete this link in the link list of the parent webpage.

The invalid link filtering method provided by the present invention is implemented based on content correlation analysis, which can make the filtered links more truly reflect the mutual relationship between web pages, make the link correlation assumptions of web pages more reasonable, and thus greatly improve the link analysis results effectiveness.

Description of drawings

The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

Fig. 1 is a schematic diagram of a news webpage intercepted on Sina;

Figure 2 shows an example of a DOM tree converted from an HTML document;

Figure 3 shows the data results after the second filtering operation;

Figure 4 shows the status of the sites where the top 100 pages are located after three rankings.

Detailed ways

The Web invalid link filtering method that the present invention proposes roughly can be divided into the operation of two parts: the first part is to utilize the text position information in the webpage, by statistical method, removes links such as irrelevant advertisement, navigation in the webpage; The second part is On the basis of the first part, carry out a correlation analysis on the content of the web page and the content of the web page pointed to by the link, and remove those links whose content is irrelevant. Detailed descriptions are given below respectively.

1. Filtering based on text position

At present, most of the webpages are created through a unified template, and for general webpages, links related to topics are placed under the text of a webpage by the webpage creator, so this part of the filtering work is based on this assumption of. The filtering work includes first converting the HTML document into a DOM tree structure, and then searching for the smallest subtree containing the main content and links related to the theme in the DOM tree, obtaining the required link information, analyzing and removing the theme in the webpage for subsequent content Prepare for irrelevant links.

DOM (Document Object Model) is the Document Object Model, a standard interface specification formulated by W3C, and an application programming interface (API) for HTML and XML documents. After the HTML document is parsed, it is transformed into a DOM tree structure. Each node of the DOM tree is an object, and the content in the HTML document is completely contained in each node. Figure 2 is an example of a DOM tree converted from an HTML document. In a specific embodiment of the present invention, a CyberNeko HTML Parser parser is used to parse the HTML document and generate a DOM tree.

As shown in Figure 1, an HTML page can be divided into different areas. For a web page, the main part (the third area in Figure 1) is dominated by text, while the number of links in other parts is relatively large. Based on such phenomenon, the concept of link ratio (Link ratio) is defined in the present invention:

Linkratio(b _i )=LinkCount(b _i )/ContentLength(b _i )

(1)

Wherein, LinkCount(bi) represents the number of links in the i-th block, and ContentLength(bi) represents the length of non-link content in the i-th block. The threshold (th) of the link ratio is set, and when the link ratio of a certain block is smaller than the threshold, this block is considered as the main part of the web page.

The present invention is aimed at the DOM tree structure, uses block nodes in the DOM tree to divide the DOM tree into various subtrees, calculates the link ratio in each subtree, compares it with the threshold value, and if it is less than the threshold value, then sets the block as the main body block , and then look back to find the nearest parent block node that contains the block, take this node as the target node, and output the link in the node as the basis for subsequent analysis.

Since the selection of block nodes determines the granularity of block web pages, the present invention selects table (div) and tr nodes as block nodes preferentially through experiments.

2. Removing unrelated links based on text content

In addition to the structure of the link in the web page, the text content of the link itself also provides a lot of information for the analysis of the web page, including the entry text of the link, the title of the web page pointed to by the link, and the main content. The content similarity between web page texts can be used to analyze whether the link has recommendation significance. Therefore, it is necessary to use this information to re-filter links to remove links that are not relevant to the topic.

Compared with other documents, web page text has limited structure, or even if it has some form of structure, it focuses on format rather than text content, and the structure of different types of content is inconsistent; in addition, the content of text is natural language In addition to the method of full-text matching, it is difficult for a computer to judge the content similarity between the two. Therefore, before analyzing the content similarity of two (or more) web pages, it is necessary to preprocess the text content of the web pages, extract the main content that can represent the two (or more) texts, and then compare them .

For the representation of text content, it is necessary to define a clear text model to form a representation that can be processed by a computer. Generally speaking, text models can be divided into three categories: Boolean model, probability model and vector space model (Vector Space Model), Among them, the vector space model is a widely used text model, which is more accurate than the Boolean model, and does not require the learning process of the probability model. Therefore, the present invention uses a vector space model to analyze text similarity.

The process of performing text preprocessing on the main content in all web pages includes the following steps:

1. Text segmentation

For a text, a word is the smallest meaningful language component that can act independently. Since English words use spaces as natural delimiters, while Chinese uses characters as the basic writing unit, and there is no obvious distinction between words. Therefore, Chinese word analysis is the basis and key of Chinese information processing.

The present invention processes each content through the text segmentation technology, and the purpose is to distinguish each word in the content, so as to reflect the content of the text through the word set (bag-of-word) representation of the text, and form the content of the text set. feature vector for subsequent processing.

2. Statistical word frequency

For subsequent calculations, a large number of vocabulary needs to be counted, so it is necessary to count the occurrence frequency of each word in the vocabulary set segmented from each text and save it.

3. Calculate TF-IDF (term frequency-inverse document frequency, term frequency-inverse document frequency) vector

In the vector space model, the TF-IDF vector representation is used. The TF-IDF vector reflects the word space of the text collection, and each of its vector components corresponds to a word. The specific definition of TF-IDF is:

d(i)=TF-IDF(i)=TF(W _i ,Doc)*IDF(W _i )=TF(W _i ,Doc)*log(D/DF(W _i )) (2)

Among them, TF(Wi,Doc) is the frequency of occurrence of the word Wi in the text Doc, D is the total number of texts, and DF(Wi) is the number of texts in which the word Wi appears at least once in the total text set.

4. Form a vector space

After the feature subset of the text is extracted, the vector space model corresponding to the text set can be established. In the vector space model, the text space is regarded as a vector space composed of a set of orthogonal term vectors.

Each text is represented as one of the feature vectors:

V(d)=(t ₁ , w ₁ (d); . . . ; t _i , w _i (d); . . . ; t _n , w _n (d))

(3)

Among them, ti is the i-th word in text d, and wi(d) is the weight of ti for text d.

wi(d) is generally defined as a function of the frequency tfi(d) of ti appearing in the text d. The present invention uses the value of TF-IDF as a feature vector, that is, uses the calculated TF-IDF value of words as wi(d) in the vector.

After obtaining the feature vector of the text, the feature vector can be used to calculate the content similarity between two (or more) texts, and use the content similarity to remove irrelevant links in the web page.

There are many methods for calculating content similarity by using feature vectors, and the present invention mainly adopts two methods, one is to consider the degree of overlap of terms contained in two feature vectors. Define text similarity as:

$\mathrm{<span\; class="notranslate">\; sim</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; no</span>}}_{<span\; class="notranslate">\; \&cap;</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>}{{\mathrm{<span\; class="notranslate">\; no</span>}}_{<span\; class="notranslate">\; \&cup;</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$

Among them, sim(di, dj) represents the text similarity between text di and dj, n _∩ (d _i , d _j ) is the same feature vector V(di) and V(dj) of text di and dj have the same The number of entries, n _∪ (d _i , d _j ) is the number of all entries that V(di) and V(dj) have.

The other is a method that considers the cosine of the angle between the two eigenvectors. Define text similarity as:

$\mathrm{<span\; class="notranslate">\; sim</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; V</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; V</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>}{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; V</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; *</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; V</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>}<span\; class="notranslate">\; =</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; im</span>}}<span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; jm</span>}}}{\sqrt{<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; im</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span><span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; jm</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; )</span>$

Among them, sim(di, dj) represents the text similarity between text di and dj, V(di) is the feature vector of text di, and wim represents the TF-IDF value of word tm in text di.

Regarding the problem of removing invalid links, the content similarity analysis for web documents and ordinary documents is different, because the former has link information that can be used. In fact, both the entry text of the link and the corresponding web page title can be used for content similarity analysis. These texts are much shorter than the body of the web page and, to a large extent, are a brief summary of the body of the web page.

Based on this idea, the present invention uses a progressive method to judge, and performs three-layer analysis in the processing process. If the texts obtained at the current layer are subject-related, they are considered to be related in content, and the following calculations are no longer performed. ; If the obtained content is irrelevant, proceed to the next layer of analysis. Only when all three layers get irrelevant results are they considered content irrelevant and the link is removed from the list.

The first layer: content correlation analysis based on the entry text

For links in web pages, the easiest thing to get is the link's entry text. The link entry text is a summary of the pointed web page by the creators of other web pages, and it is valuable to use the entry text to replace the original content of the web page. Because the length of the link entry text is generally limited, the computational complexity of the text similarity will not be very large.

However, the number of words contained in the entry text is limited, and the dimension will be different when the vector is created, and the cosine of the angle between the two vectors cannot be calculated, so the first calculation method can only be used. In a calculation example, the inventor only selects words with the top ten TF-IDF values in the parent webpage as the feature vectors of the parent webpage, and uses the first method to obtain the text similarity between the webpage pointed to by the link and the parent webpage , and then compared with the threshold, if it is greater than the threshold, they are considered to have content relevance, and if it is less than the threshold, continue to the calculation of the next layer.

The second layer: Carry out content correlation analysis according to the title of the web page

For general webpage builders, they will give their webpage a short title, and in the HTML document of the webpage, use the title tag to indicate that the title of the usual webpage is a description of the webpage, and usually includes some content of the text in the webpage. Important keywords, using such a title can also replace the text in the web page to analyze the similarity of the content, but the title of the web page is added to the web page by the web page creator himself, with the subjectivity of the web page creator, some web pages The creator will add a lot of irrelevant words to the title of the webpage to improve his search engine ranking and increase the number of clicks on the webpage. Therefore, when analyzing the title, he puts the analysis of the title after the entry text.

The correlation calculation method for the title is consistent with the link entry text.

The third layer: Carry out content correlation analysis according to the content of the webpage text

Because the amount of data in the text of the webpage is large, in the calculation process, only the negative results obtained by the first two layers will be analyzed at this layer, because in this layer, the feature vectors of the text can have the same dimension (even if there are no words, it can be processed by adding words and setting the TF-IDF value to 0), so that the second calculation method can be used to obtain the content similarity between two web pages.

If two (or more) webpage topics are found to be irrelevant at these three levels, then delete this link in the link list of the parent webpage.

Next, a series of experimental data will be used to illustrate the effect of the invalid link filtering method based on content correlation proposed by the present invention, and the improvement effect of the link analysis algorithm will be illustrated by comparing the PageRank value of the page before and after removal.

The experimental data set used in the present invention is randomly obtained from CWT200g. The inventor aimed at the 627,036 hosts providing web services in China found in the webpage collection in November 2005, obtained 88,303 webpages by eliminating duplicate websites and spam websites, and collected the webpages of these webpages. For 3, the amount of data collected by a single website is not limited, the initial data set is obtained, and then the deduplication of web pages is performed to obtain a collection of non-repeated web pages. According to the size of the website reflected in the webpage collection, sampling is carried out, and finally the CWT200g test set with a capacity of 197GB is obtained.

The present invention randomly extracts 1,524,077 webpages from 1,421 websites in CWT200g, removes incomplete preservation, and only keeps html, htm, xml, jsp, asp types of webpages, and finally obtains 1,427,001 webpages as a follow-up experiment data set.

Table 1 is some statistical data for the experimental data set, where the out-degree of the web page is the number of links in the web page, the internal links are the number of links in the web page pointing to the web pages in the experimental data set, and the adjusted out-degree is zero The number represents the number of newly obtained web pages with an out-degree of 0 after the external links in the data set are removed.

reference variable Quantity (unit: piece) Number of websites 1421 Number of pages 1427001 Average number of pages on the site 1004.2 number of links 80457759 average number of links per page 56.38 The number of pages with an out-degree of 0 65256 internal link 27312578 Internal Link Ratio 33.95% A net with an adjusted outdegree of 0 220122 Number of pages

Table 1

The present invention uses the above-mentioned two-step operation to remove invalid links. Since the two-step operations deal with different types of invalid links, the ratios of links removed by the two-step operations are also very different.

After performing two-step filtering, the present invention counts the proportion of links removed in each step for different types of web pages. Figure 3 is the data result after the second step of filtering. It can be seen from Figure 3 that the filtering ratio of the link in the first step of filtering is very high, which is consistent with the previous analysis, and there are a lot of advertising links in com-like web pages, so in this After one-step filtering, the percentage of removed links is the highest. But at the same time, it can also be seen that for education and government web pages, according to the convention, these websites will not include a large number of advertising links, but it can be seen from Figure 3 that the removal ratio of these two types of web pages is also very high High, through the observation of some webpages in the data set, it is found that most of the links in these two types of webpages are concentrated on both sides of the webpage, and the links in this part are mainly classified as "popular" or "latest". Links are the main link, but for the evaluation of the relevance of the topic, these links are also invalid links, so from the perspective of link analysis, they should also be filtered out from the links.

It can be seen from Figure 3 that after the two-step filtering operation, the proportion of the remaining effective links is basically the same as that obtained by manual evaluation. In fact, the filter conditions set in this embodiment are relatively loose, so the proportion of effective links obtained is slightly higher than that obtained by manual evaluation.

A search engine is essentially a recommendation system, and it should recommend as many high-quality websites as possible to users. Therefore, for the ranking algorithm, try to avoid a large number of top-ranked pages from the same website, otherwise it will seriously affect the diversity of recommendations. From this point of view, the improvement effect of the invalid link filtering method provided by the present invention on PageRank will be further analyzed.

Figure 4 compares the sites where the top 100 pages are located after three rankings. Among them, baseline represents the ranking result without any processing, first represents the result after only the first step of filtering, and second represents the result after two steps of filtering. It can be seen from Figure 4 that the number of websites included in the top 100 of PageRank after the first step is 46, and after the second step of filtering, there are 69 websites included in the top 100. After the second step of filtering, the number of sites included in the top 100 has increased significantly, indicating that through this step of filtering, more valuable websites can be found and more valuable choices can be provided to users.

The method for filtering invalid Web links based on content correlation provided by the present invention has been described in detail above. For those skilled in the art, any obvious changes made to it without departing from the essence of the present invention will constitute an infringement of the patent right of the present invention and will bear corresponding legal responsibilities.

Claims (10)

1. a kind of Web invalid link filtering method based on content relevance, it is characterized in that comprising the steps: (1) Use the text position information in the webpage to remove irrelevant advertising links and navigation links in the webpage through statistical methods; (2) Carry out correlation analysis on the content of the web page and the content of the web page pointed to by the link, and remove invalid links with irrelevant content. 2. Web invalid link filtering method as claimed in claim 1, is characterized in that: In the step (1), the HTML document is first converted into a DOM tree structure, and then the smallest subtree containing the main content and links related to the topic is searched in the DOM tree structure to obtain the required link information. 3. Web invalid link filtering method as claimed in claim 2, is characterized in that: Use CyberNeko HTML Parser to convert HTML documents into DOM tree structures. 4. Web invalid link filtering method as claimed in claim 2, is characterized in that: For the DOM tree structure, first utilize the block node to divide the DOM tree into each subtree, calculate the link ratio in each subtree, and compare it with a predetermined threshold; if it is less than the threshold, then the block is set as the main body block, Then backtrack to find the nearest parent block node that contains the block, take the parent block node as the target node, and output the link in the parent block node as the basis for subsequent analysis. 5. Web invalid link filtering method as claimed in claim 4, is characterized in that: When selecting block nodes, table (div) and tr nodes are preferred as block nodes. 6. the Web invalid link filtering method as claimed in claim 1, is characterized in that: In the step (2), before performing the content correlation analysis of the webpage, the text of the webpage is preprocessed, and the content representing each text is extracted for comparison. 7. Web invalid link filtering method as claimed in claim 6, is characterized in that: The process of text preprocessing includes the following steps: First perform text segmentation, then count the frequency of words in the text, calculate the TF-IDF vector, and form a vector space model corresponding to the text set; use the feature vector of the text to calculate the content similarity between each text, and use the content similarity degree to remove irrelevant links in web pages. 8. Web invalid link filtering method as claimed in claim 7, is characterized in that: The content similarity is determined by the degree of overlap of terms contained in the feature vectors of each text. 9. Web invalid link filtering method as claimed in claim 7, is characterized in that: The content similarity is determined by the cosine of the included angle among the feature vectors of each text. 10. Web invalid link filtering method as claimed in claim 1, is characterized in that: In described step (2), described content correlation analysis comprises three layers of operations: the first layer is to carry out content correlation analysis according to entry text; The second layer is to carry out content correlation analysis according to the title of webpage; The third layer is Carry out content correlation analysis according to the content of the webpage body; if the conclusion that the theme of the webpage is irrelevant is obtained in these three layers, delete this link in the link list of the parent webpage.

Images