Adds the "driftnet" source. #1612
Conversation
WalkthroughA new passive subdomain enumeration source named "driftnet" has been integrated. This includes its implementation, registration in the list of available sources, and updates to relevant test expectations. The "driftnet" source supports concurrent API queries, recursive enumeration, and requires API keys. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant PassiveEnum
participant DriftnetSource
participant DriftnetAPI
User->>PassiveEnum: Start subdomain enumeration
PassiveEnum->>DriftnetSource: Run(domain, session)
DriftnetSource->>DriftnetAPI: Query multiple endpoints (concurrently)
DriftnetAPI-->>DriftnetSource: Return subdomain data (JSON)
DriftnetSource->>PassiveEnum: Send unique subdomain results
PassiveEnum->>User: Aggregate and return results
Suggested reviewers
Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
v2/pkg/passive/sources.go(2 hunks)v2/pkg/passive/sources_test.go(3 hunks)v2/pkg/subscraping/agent.go(1 hunks)v2/pkg/subscraping/sources/driftnet/driftnet.go(1 hunks)
🧰 Additional context used
🧠 Learnings (2)
v2/pkg/passive/sources.go (1)
Learnt from: x-stp
PR: projectdiscovery/subfinder#0
File: :0-0
Timestamp: 2025-06-24T16:03:30.467Z
Learning: When fixing HTTP response handling bugs in subfinder sources, the correct pattern is to use `defer session.DiscardHTTPResponse(resp)` after successful requests to ensure the response body remains open for reading, and call `session.DiscardHTTPResponse(resp)` immediately in error cases.
v2/pkg/subscraping/agent.go (1)
Learnt from: x-stp
PR: projectdiscovery/subfinder#0
File: :0-0
Timestamp: 2025-06-24T16:03:30.467Z
Learning: When fixing HTTP response handling bugs in subfinder sources, the correct pattern is to use `defer session.DiscardHTTPResponse(resp)` after successful requests to ensure the response body remains open for reading, and call `session.DiscardHTTPResponse(resp)` immediately in error cases.
🧬 Code Graph Analysis (2)
v2/pkg/passive/sources.go (2)
v2/pkg/subscraping/sources/driftnet/driftnet.go (1)
Source(24-30)v2/pkg/subscraping/types.go (1)
Source(37-62)
v2/pkg/subscraping/sources/driftnet/driftnet.go (2)
v2/pkg/subscraping/types.go (2)
Session(71-78)Statistics(29-34)v2/pkg/subscraping/utils.go (1)
PickRandom(12-20)
🔇 Additional comments (4)
v2/pkg/passive/sources.go (1)
29-29: LGTM!The driftnet source is correctly imported and registered in the AllSources array, following the established pattern.
Also applies to: 74-74
v2/pkg/passive/sources_test.go (1)
29-29: Test data correctly updated.The inclusion of "driftnet" in all three test arrays aligns with the source implementation returning
truefor bothIsDefault()andHasRecursiveSupport().Also applies to: 76-76, 106-106
v2/pkg/subscraping/agent.go (1)
140-140: Confirm global acceptance of HTTP 204 responsesThe update to agent.go now treats 204 (No Content) as successful for all subscraping sources, not just driftnet. That means any source that only expected 200 OK will now see a “successful” response on 204 and proceed to parse an empty body.
Please verify that none of these endpoints legitimately return 204, or else restrict 204-acceptance to driftnet only. Impacted sources include (but may not be limited to):
- v2/pkg/subscraping/sources/threatcrowd/threatcrowd.go
- v2/pkg/subscraping/sources/pugrecon/pugrecon.go
- v2/pkg/subscraping/sources/leakix/leakix.go
- v2/pkg/subscraping/sources/netlas/netlas.go
- v2/pkg/subscraping/sources/digitalyama/digitalyama.go
- v2/pkg/subscraping/sources/anubis/anubis.go
If any of these APIs can send a 204, ensure their code handles an empty response body correctly or scope the 204 check to driftnet.
v2/pkg/subscraping/sources/driftnet/driftnet.go (1)
60-184: Well-implemented source with proper concurrency and deduplication.The implementation demonstrates good practices:
- Concurrent requests to multiple endpoints with proper synchronization
- Deduplication across endpoints using
sync.Map- Comprehensive error handling and statistics tracking
- Proper cleanup in all code paths
|
@dogancanbakir Is it likely to be safe to allow 204 responses globally, as this PR currently proposes to do? If you think it isn't safe, it would be okay to leave that section of code as-is. The only issue would be a warning when one of Driftnet's endpoints fails to return any results and sends code 204. |
|
Test failure here seems unrelated: I'm also seeing the same failure on the |
Updates the httpRequestWrapper() function to allow 204 responses.
172cdb9 to
8f5680e
Compare
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
pkg/subscraping/sources/driftnet/driftnet.go (2)
85-86: Fix typo in comment.- // Return the result c + // Return the results channel
124-124: Add description to the incomplete comment.-// runSubsource +// runSubsource queries a specific driftnet endpoint for subdomains and sends results to the channel
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
pkg/passive/sources.go(2 hunks)pkg/passive/sources_test.go(3 hunks)pkg/subscraping/agent.go(1 hunks)pkg/subscraping/sources/driftnet/driftnet.go(1 hunks)
✅ Files skipped from review due to trivial changes (2)
- pkg/passive/sources.go
- pkg/passive/sources_test.go
🧰 Additional context used
🧠 Learnings (1)
pkg/subscraping/agent.go (1)
Learnt from: x-stp
PR: projectdiscovery/subfinder#0
File: :0-0
Timestamp: 2025-06-24T16:03:30.467Z
Learning: When fixing HTTP response handling bugs in subfinder sources, the correct pattern is to use `defer session.DiscardHTTPResponse(resp)` after successful requests to ensure the response body remains open for reading, and call `session.DiscardHTTPResponse(resp)` immediately in error cases.
🧬 Code Graph Analysis (1)
pkg/subscraping/sources/driftnet/driftnet.go (2)
pkg/subscraping/types.go (2)
Session(71-78)Statistics(29-34)pkg/subscraping/utils.go (1)
PickRandom(12-20)
|
@dogancanbakir I think the bot has resolved the query I had on 204s, and the tests are passing for me now. |
|
@0x4500 Thanks for the PR! I'll review it asap. |
|
@0x4500 Is this expected? $ go run . -d vulnerabletarget.com -s driftnet -v
__ _____ __
_______ __/ /_ / __(_)___ ____/ /__ _____
/ ___/ / / / __ \/ /_/ / __ \/ __ / _ \/ ___/
(__ ) /_/ / /_/ / __/ / / / / /_/ / __/ /
/____/\__,_/_.___/_/ /_/_/ /_/\__,_/\___/_/
projectdiscovery.io
[INF] Current subfinder version v2.8.0 (latest)
[INF] Loading provider config from /Users/dogancanbakir/Library/Application Support/subfinder/provider-config.yaml
[DBG] API key(s) found for driftnet.
[DBG] Selected source(s) for this search: driftnet
[INF] Enumerating subdomains for vulnerabletarget.com
[driftnet] www.vulnerabletarget.com
- [DBG] Response for failed request against https://api.driftnet.io/v1/domain/rdns?host=vulnerabletarget.com&summarize=host&summary_context=dns-ptr&summary_limit=10000:
- [DBG] Response for failed request against https://api.driftnet.io/v1/scan/protocols?field=host:vulnerabletarget.com&summarize=host&summary_context=cert-dns-name&summary_limit=10000:
www.vulnerabletarget.com
[INF] Found 1 subdomains for vulnerabletarget.com in 828 milliseconds 654 microseconds |
@dogancanbakir Yeah, this is the 204 issue I was mentioning above. In detail, the issue is that the Driftnet API legitimately responds with a 204 (No Content) response if it has no results from one of its applicable endpoints. This is not intended to indicate an error, just a lack of results, and there might still be results from one of the other Driftnet endpoints (as there are in your test). However, the code in I did initially update I'm not really sure how to handle this issue properly within the subfinder framework, and your advice would be very welcome! As things stand with this PR, it all seems to work but you do get the error you see in debug mode. |
|
Got it, thanks! Let's leave it as is for now. I'll open another issue to track it. Tracking at #1620 |
This PR adds driftnet.io as a default source.
Driftnet has multiple endpoints which can return subdomains, and so internally the code for this source makes multiple HTTP requests and combines the results. Results are de-duplicated so that the "driftnet" source does not return the same subdomain multiple times.
The Driftnet API can legitimately return an HTTP 204 code when there are no results. To prevent this causing an error, the
httpRequestWrapper()function insidev2/pkg/subscraping/agent.gois updated to handle 204 responses in the same way as 200 responses are currently handled.Summary by CodeRabbit
New Features
Tests