这是indexloc提供的服务,不要输入任何密码
Skip to content

[FEAT]: Website scraping depth #1190

@shatfield4

Description

@shatfield4

What would you like to see?

We should create a data connector that allows scraping websites with a configurable depth

  • Should scrape all <a> tags on the site and look for all links that match the original domain name then scrape X levels deep
  • Should all be done using the existing puppeteer instance that is already used for scraping links

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions