Adding url uploads to document picker #375
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
collector/scripts/link.pyIn the def process_single_link(url):
if not url:
return False, "Invalid URL!", None
session = HTMLSession()
req = session.get(url)
if not req.ok:
return False, "Could not reach this URL.", None
req.html.render()
with tempfile.NamedTemporaryFile(mode = "w") as tmp:
tmp.write(req.html.html)
tmp.seek(0)
loader = UnstructuredHTMLLoader(tmp.name)
data = loader.load()[0]
full_text = data.page_content
if not full_text:
return False, "Could not parse any meaningful data from this URL.", None
link_meta = append_meta(req, full_text, True)
token_count = len(tokenize(full_text))
link_meta['pageContent'] = full_text
link_meta['token_count_estimate'] = token_count
return True, None, link_metaIn the def process_single_link(url):
if not url:
return False, "Invalid URL!", None
session = HTMLSession()
req = session.get(url)
if not req.ok:
return False, "Could not reach this URL.", None
req.html.render()
html_content = req.html.html
with tempfile.NamedTemporaryFile(mode = "w") as tmp:
tmp.write(html_content)
tmp.seek(0)
loader = UnstructuredHTMLLoader(tmp.name)
data = loader.load()[0]
full_text = data.page_content
if not full_text:
return False, "Could not parse any meaningful data from this URL.", None
link_meta = append_meta(req, full_text, True)
token_count = len(tokenize(full_text))
link_meta['pageContent'] = full_text
link_meta['token_count_estimate'] = token_count
return True, None, link_meta |
no autocomplete on URLs
cabwds
pushed a commit
to cabwds/anything-llm
that referenced
this pull request
Jul 3, 2025
* WIP adding url uploads to document picker * fix manual script for uploading url to custom-documents * fix metadata for url scraping * wip url parsing * update how async link scraping works * docker-compose defaults added no autocomplete on URLs --------- Co-authored-by: timothycarambat <rambat1010@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolves #363
Adds an input for users to put links to scrape and insert them into their documents