Add YouTube Transcript Pulling to scrapeGenericUrl & Improve Web Scraper Tool Introspection
#4537
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Introduced functionality to handle YouTube URLs by validating them and fetching video transcripts. - Updated the `processVia` logic to include a new option for processing YouTube video transcripts. - Enhanced the scraping function to format and return transcript content as a document if required. - Added a utility function to validate YouTube URLs.
…ed references in the generic URL scraper.
…, ensuring all references are consistent in the URL validation tests.
10 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Type
Relevant Issues
resolves #4508
What is in this change?
This PR introduces two main changes:
scrapeGenericUrlfunction.What These Changes Enable
YouTube Transcript Support
With these updates, chatting with an LLM using
@agentcan now automatically fetch a YouTube transcript when given a video URL via theweb_scrapertool.Example usage:
@agent Please summarize this video https://www.youtube.com/watch?v=B_H1DxOI6XsAdditionally, users can now pass a YouTube video URL directly into the URL input field within the RAG document modal to create a document from that video, effectively bypassing the need for the dedicated YouTube data connector.
Improved Introspection Logging
When
@agentcalls theweb_scrapertool and passes in a URL. The tool first verifies what kind of resource it is by analyzing the URL itself and making a HEAD call to retrieve itsContent-Typeheader. Based on this information the introspection logs will inform the user whether the tools will begin toAdditional Information
Developer Validations
yarn lintfrom the root of the repo & committed changes