New source: DomainsProject #1663

tb0hdan · 2025-10-20T20:31:18Z

New source: DomainsProject
Requires credentials
Limit ~ 10 domains / 1m

Summary by CodeRabbit

New Features
- Added DomainsProject as a new passive subdomain data source.
- Requires DomainsProject API credentials to enable the source.
- Exposes per-source statistics (results, errors, time taken, skipped).
Tests
- Test expectations updated to include the new DomainsProject source.

coderabbitai · 2025-10-20T20:31:33Z

Walkthrough

A new passive subdomain source "domainsproject" is added and registered; it implements a Source that uses BasicAuth to query the DomainsProject API, decodes JSON responses, emits subdomain results, and tracks simple scrape statistics.

Changes

Cohort / File(s)	Summary
Source Registration `pkg/passive/sources.go`	Import `domainsproject` package and register `&domainsproject.Source{}` in `AllSources`.
DomainsProject Source Implementation `pkg/subscraping/sources/domainsproject/domainsproject.go`	New `Source` type implementing `Run(ctx, domain, session)` with random API-key selection, authenticated GET, JSON decoding of `{ domains, error }`, emission of `subscraping.Result` (subdomains/errors/skipped), and `Statistics()` reporting.
Tests Updated `pkg/passive/sources_test.go`	Add `"domainsproject"` to `expectedAllSources` and `expectedDefaultSources` in `TestSourceCategorization`.

Sequence Diagram

sequenceDiagram
    participant Client
    participant Source
    participant DomainsProjectAPI as DomainsProject API
    participant Results

    Client->>Source: Run(ctx, domain, session)
    activate Source
    Source->>Source: pickRandomApiKey()
    alt no API key
        Source->>Results: emit skipped result
    else API key present
        Source->>DomainsProjectAPI: GET /domains?domain=... (BasicAuth)
        activate DomainsProjectAPI
        DomainsProjectAPI-->>Source: 200 JSON {domains: [...]} / {error: "..."}
        deactivate DomainsProjectAPI

        alt response.error present
            Source->>Results: emit Error result
        else valid domains list
            loop for each domain
                alt domain doesn't start with "."
                    Source->>Results: emit Subdomain result
                end
            end
        end
        Source->>Results: emit Statistics
    end
    deactivate Source
    Results-->>Client: results channel receives events

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

Poem

🐰 I hopped to a new API door,
With keys in paw and JSON to store,
Domains returned in a jittery stream,
I nibble results, track time like a dream,
Hooray — another source on the shore! ✨

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check	✅ Passed	The pull request title "New source: DomainsProject" directly and specifically describes the main change in the changeset. The PR adds a new passive subdomain scraping source called DomainsProject, which is exactly what the title communicates. The title is concise, clear, and matches the stated PR objective to add a new source. While the title doesn't cover every detail (such as credential requirements or rate limits), this is expected and appropriate for a title—those details belong in the description. The title is specific enough that a teammate scanning history would immediately understand that this PR introduces a new source integration.
Docstring Coverage	✅ Passed	No functions found in the changes. Docstring coverage check skipped.

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c1b64d8 and 3ba0a30.

📒 Files selected for processing (1)

pkg/passive/sources_test.go (2 hunks)

🔇 Additional comments (2)

pkg/passive/sources_test.go (2)

29-29: LGTM: Correctly added to all sources list.

The "domainsproject" source is correctly inserted in alphabetical order between "dnsrepo" and "driftnet".

77-77: Code change is correct.

The addition of "domainsproject" to expectedDefaultSources is appropriate because the source's IsDefault() method returns true. The codebase already includes many default sources that require API credentials (chaos, censys, shodan, c99, etc.).

Regarding rate limits: the ~10 domains/minute mentioned in the PR description is an external API service constraint, not implemented in code. No rate limiting logic exists in the domainsproject source file or in other default sources, confirming this is handled at the API service level rather than in subfinder. This is a product-level consideration, not a code defect.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 1

🧹 Nitpick comments (2)

pkg/subscraping/sources/domainsproject/domainsproject.go (2)
84-84: Use %s format specifier for string.

Using %v for a string value is non-idiomatic. Use %s instead, or directly wrap with errors.New().

Apply this diff:
-			Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf("%v", response.Error),
+			Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf("%s", response.Error),
90-95: Consider adding a comment for the subdomain filtering logic.

The filtering of subdomains starting with "." could benefit from a brief comment explaining why these are excluded (e.g., wildcard entries or malformed data from the API).

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 2ba3645 and 02ea9f8.

📒 Files selected for processing (2)

pkg/passive/sources.go (2 hunks)
pkg/subscraping/sources/domainsproject/domainsproject.go (1 hunks)

🧰 Additional context used

🧬 Code graph analysis (2)

pkg/passive/sources.go (1)

pkg/subscraping/sources/domainsproject/domainsproject.go (1)

Source (16-22)

pkg/subscraping/sources/domainsproject/domainsproject.go (2)

pkg/subscraping/types.go (3)

Session (71-78)

BasicAuth (23-26)

Statistics (29-34)

pkg/subscraping/utils.go (2)

PickRandom (12-20)

CreateApiKeys (22-30)

🔇 Additional comments (6)

pkg/passive/sources.go (2)

28-28: LGTM!

The import is correctly positioned in alphabetical order and follows the project's import path conventions.

75-75: LGTM!

The new source is correctly registered in alphabetical order within the AllSources array, consistent with the existing pattern.

pkg/subscraping/sources/domainsproject/domainsproject.go (4)

1-13: LGTM!

Package declaration and imports are clean and appropriate for the source implementation.

15-32: LGTM!

The struct definitions are well-designed and align with the subscraping framework patterns. The response struct appropriately handles both success and error cases from the API.

35-50: LGTM!

The initialization and API key selection logic correctly follows the established subscraping patterns. The skipped flag is properly set when credentials are unavailable.

101-131: LGTM!

All interface methods are correctly implemented:

Returns appropriate source name

Correctly indicates it needs API keys (matching PR description)

Properly uses the CreateApiKeys utility for username/password pairs

Statistics method correctly maps all tracked metrics

pkg/subscraping/sources/domainsproject/domainsproject.go

ehsandeep · 2025-10-21T15:08:15Z

@coderabbitai create and link github issue to this PR

coderabbitai · 2025-10-21T15:09:02Z

Tip

For best results, initiate chat on the files or code changes.

@ehsandeep: I'd be happy to create and link a GitHub issue to this PR. Could you please provide more details about what the issue should contain? For example:

What should the issue title and description be?
Is this for tracking a specific concern or follow-up task related to this PR?
Should it reference any specific aspects of the DomainsProject source implementation?

Once you provide these details, I'll create the issue and link it to this PR.

dogancanbakir · 2025-10-22T04:29:28Z

Thanks for the PR and for your contributions to the OSS!

I'm trying to get an API key, but ran into an error (http 502) when accessing the API subdomain. Could you add instructions on how to get one?

tb0hdan · 2025-10-22T18:03:04Z

Yes, this is a private API that I offer to my subscribers through Patreon: https://patreon.com/tb0hdan -> Memberships
or a direct link: https://www.patreon.com/checkout/tb0hdan?rid=9257515
Credentials are issued manually upon subscription and consist of a login and a password.

@dogancanbakir I've sent you test credentials to your email that I've extracted from your GitHub account.
Confirm receipt.

dogancanbakir · 2025-10-30T08:36:06Z

@tb0hdan Thanks for the info! I've tried with credentials you sent over email but it didn't work.

$ go run . -s domainsproject -d vulnerabletarget.com -v

               __    _____           __         
   _______  __/ /_  / __(_)___  ____/ /__  _____
  / ___/ / / / __ \/ /_/ / __ \/ __  / _ \/ ___/
 (__  ) /_/ / /_/ / __/ / / / / /_/ /  __/ /    
/____/\__,_/_.___/_/ /_/_/ /_/\__,_/\___/_/

                projectdiscovery.io

[INF] Current subfinder version v2.9.0 (latest)
[INF] Loading provider config from /Users/dogancanbakir/Library/Application Support/subfinder/provider-config.yaml
[DBG] API key(s) found for domainsproject.
[DBG] Selected source(s) for this search: domainsproject
[INF] Enumerating subdomains for vulnerabletarget.com
[DBG] Response for failed request against https://api.domainsproject.org/api/tld/search?domain=vulnerabletarget.com:
Unauthorized
[WRN] Encountered an error with source domainsproject: unexpected status code 401 received from https://api.domainsproject.org/api/tld/search?domain=vulnerabletarget.com
[INF] Found 0 subdomains for vulnerabletarget.com in 344 milliseconds 351 microseconds

tb0hdan · 2025-10-30T17:40:03Z

Had some wires crossed there. Fixed.

dogancanbakir · 2025-11-03T12:11:42Z

This time I got 500:

$ go run . -s domainsproject -d vulnerabletarget.com -v

               __    _____           __         
   _______  __/ /_  / __(_)___  ____/ /__  _____
  / ___/ / / / __ \/ /_/ / __ \/ __  / _ \/ ___/
 (__  ) /_/ / /_/ / __/ / / / / /_/ /  __/ /    
/____/\__,_/_.___/_/ /_/_/ /_/\__,_/\___/_/

                projectdiscovery.io

[INF] Current subfinder version v2.9.0 (latest)
[INF] Loading provider config from /Users/dogancanbakir/Library/Application Support/subfinder/provider-config.yaml
[DBG] API key(s) found for domainsproject.
[DBG] Selected source(s) for this search: domainsproject
[INF] Enumerating subdomains for vulnerabletarget.com
[DBG] Response for failed request against https://api.domainsproject.org/api/tld/search?domain=vulnerabletarget.com:
no subdomains found for vulnerabletarget.com
[WRN] Encountered an error with source domainsproject: unexpected status code 500 received from https://api.domainsproject.org/api/tld/search?domain=vulnerabletarget.com
[INF] Found 0 subdomains for vulnerabletarget.com in 4 seconds 141 milliseconds

tb0hdan · 2025-11-03T15:37:14Z

That was by design: "no subdomains found for vulnerabletarget.com". I've updated the API to return an empty domains list and 200 instead.

New source: DomainsProject

02ea9f8

coderabbitai bot reviewed Oct 20, 2025

View reviewed changes

pkg/subscraping/sources/domainsproject/domainsproject.go Show resolved Hide resolved

tb0hdan added 2 commits October 20, 2025 16:36

CR fixes

c1b64d8

Add domainsproject to sources tests

3ba0a30

ehsandeep requested a review from dogancanbakir October 21, 2025 15:07

dogancanbakir approved these changes Nov 4, 2025

View reviewed changes

dogancanbakir merged commit 6c1e538 into projectdiscovery:dev Nov 4, 2025
10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

New source: DomainsProject #1663

New source: DomainsProject #1663

Uh oh!

tb0hdan commented Oct 20, 2025 •

edited by coderabbitai bot

Loading

Uh oh!

coderabbitai bot commented Oct 20, 2025 •

edited

Loading

Uh oh!

coderabbitai bot left a comment

Uh oh!

Uh oh!

ehsandeep commented Oct 21, 2025

Uh oh!

coderabbitai bot commented Oct 21, 2025

Uh oh!

dogancanbakir commented Oct 22, 2025

Uh oh!

tb0hdan commented Oct 22, 2025

Uh oh!

dogancanbakir commented Oct 30, 2025

Uh oh!

tb0hdan commented Oct 30, 2025

Uh oh!

dogancanbakir commented Nov 3, 2025

Uh oh!

tb0hdan commented Nov 3, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

New source: DomainsProject #1663

New source: DomainsProject #1663

Uh oh!

Conversation

tb0hdan commented Oct 20, 2025 • edited by coderabbitai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai bot commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram

Estimated code review effort

Poem

Pre-merge checks and finishing touches

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ehsandeep commented Oct 21, 2025

Uh oh!

coderabbitai bot commented Oct 21, 2025

Uh oh!

dogancanbakir commented Oct 22, 2025

Uh oh!

tb0hdan commented Oct 22, 2025

Uh oh!

dogancanbakir commented Oct 30, 2025

Uh oh!

tb0hdan commented Oct 30, 2025

Uh oh!

dogancanbakir commented Nov 3, 2025

Uh oh!

tb0hdan commented Nov 3, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tb0hdan commented Oct 20, 2025 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Oct 20, 2025 •

edited

Loading