这是indexloc提供的服务，不要输入任何密码
Skip to content

Blocks images in sitemaps from being parsed. #56

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jun 15, 2023
Merged

Blocks images in sitemaps from being parsed. #56

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ffa7453
Adds ability to import sitemaps to include a website
skidvis Jun 14, 2023
e43cac5
adds example sitemap url
skidvis Jun 14, 2023
32ab1c9
Merge branch 'master' of github.com:skidvis/anything-llm
skidvis Jun 14, 2023
ecb783d
adds filter to bypass common image formats
skidvis Jun 14, 2023
54e4a58
moves filetype ignoring to sitemap script
skidvis Jun 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 1 addition & 6 deletions collector/scripts/link.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@
from langchain.document_loaders import UnstructuredHTMLLoader
from .link_utils import append_meta
from .utils import tokenize, ada_v2_cost
from requests.exceptions import ReadTimeout

# Example Channel URL https://tim.blog/2022/08/09/nft-insider-trading-policy/
def link():
Expand Down Expand Up @@ -91,11 +90,7 @@ def links():
# parse links from array
def parse_links(links):
totalTokens = 0
for link in links:
if link.endswith(".pdf"):
print(f"Skipping PDF file: {link}")
continue

for link in links:
print(f"Working on {link}...")
session = HTMLSession()

Expand Down
14 changes: 13 additions & 1 deletion collector/scripts/sitemap.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import requests
import xml.etree.ElementTree as ET
from scripts.link import parse_links
import re

def parse_sitemap(url):
response = requests.get(url)
Expand All @@ -9,7 +10,10 @@ def parse_sitemap(url):
urls = []
for element in root.iter('{http://www.sitemaps.org/schemas/sitemap/0.9}url'):
for loc in element.iter('{http://www.sitemaps.org/schemas/sitemap/0.9}loc'):
urls.append(loc.text)
if not has_extension_to_ignore(loc.text):
urls.append(loc.text)
else:
print(f"Skipping filetype: {loc.text}")

return urls

Expand All @@ -25,3 +29,11 @@ def sitemap():

#parse links from array
parse_links(url_array)

def has_extension_to_ignore(string):
image_extensions = ['.jpg', '.jpeg', '.png', '.gif', '.bmp', '.pdf']

pattern = r'\b(' + '|'.join(re.escape(ext) for ext in image_extensions) + r')\b'
match = re.search(pattern, string, re.IGNORECASE)

return match is not None