[BUG]: Bulk Link Scraper does NOT work for localhost website #2459
Description
How are you running AnythingLLM?
AnythingLLM desktop app
What happened?
I can't use Bulk Link Scraper to scrap localhost website
I'm using windows desktop app in version v1.6.7
"Scraping website" forever.
And the localhost website looks like:
$ curl localhost:8000 -v
* Host localhost:8000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
* Trying [::1]:8000...
* Trying 127.0.0.1:8000...
* Connected to localhost (127.0.0.1) port 8000
> GET / HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/8.8.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Fri, 11 Oct 2024 17:09:23 GMT
< Last-modified: Fri, 11 Oct 2024 16:58:23 GMT
< Content-type: text/html; charset=UTF-8
< Content-length: 175
<
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8"/>
</head>
<body>
<h1>Directory listing for /</h1>
<ul>
<li><a href="test.txt">test.txt</a></li>
</ul>
</body>
</html>Logs
collector-2024-10-12.log
backend-2024-10-12.log
Are there known steps to reproduce?
Scrap website served in localhost