这是indexloc提供的服务,不要输入任何密码
Skip to content

Conversation

@donners
Copy link
Contributor

@donners donners commented Feb 4, 2020

Just a few little changes I made to help with error handling:

  • if we get non 2xx response, assume our request failed and return an error
  • add checks for specific errors returned by dnsdumpster and hackertarget (annoyingly they still return 2xx responses even in these cases)
  • add retries for waybackarchive - I found that this helped a lot in reducing the overall error rates from this source

@donners donners requested a review from Ice3man543 February 4, 2020 03:35
@Ice3man543
Copy link
Member

@donners Nice work with the PRs. These are the things i would love to see in subfinder. I'll review and merge your PR asap! Thanks for the excellent contributions.

@Ice3man543
Copy link
Member

@donners btw, i checked your fork and liked feature to show time taken by each source. Added it to subfinder in PR #223

@Ice3man543
Copy link
Member

@donners, i'm not merging this as of now. Let's aim this for the next release since it'll require a complete rework of the way we handle errors. Thanks for creating this PR.

@donners
Copy link
Contributor Author

donners commented Feb 13, 2020

@Ice3man543 awesome, thanks for looking at these changes!

No worries, in my case I'm calling Agent.EnumerateSubdomains() directly, so I probably missed some of the implications of these changes outside of that.

@ehsandeep
Copy link
Member

again thank you @donners for the PR on this, I'm closing this for now as possible changes have been already implemented.

@ehsandeep ehsandeep closed this Jun 5, 2020
@donners
Copy link
Contributor Author

donners commented Jun 5, 2020

Thanks for following up @bauthard! Could you point me to the changes that were implemented?

@donners donners deleted the improve-error-handling branch June 5, 2020 06:51
@ehsandeep
Copy link
Member

Hey, I was pointing to #220 (comment) but I will check with @Ice3man543 again if we are going to do more improvements here and will keep you updated.

@donners
Copy link
Contributor Author

donners commented Jun 5, 2020

Ah right. That comment was not related to this PR. It looks like 4xx/5xx status codes and dnsdumpster/hackertarget rate limits are still being ignored. Robust error handling is pretty important for my use case, would be great to see this handled properly in the source.

@ehsandeep
Copy link
Member

Right, so I'm going to create an issue for this for visibility.

@ehsandeep ehsandeep mentioned this pull request Jun 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants