Usage:

python main.py [-j n]

where n is the number of threads you want to run in parallel.

Crawling custom URLs is no longer supported for security and quality reasons. To submit a domain to crawl, please visit https://mwmbl.org/app/domain-submissions/new

Clone this repo, install poetry if necessary, cd into the repo and type

poetry install
poetry shell

then the main.py command as documented above.