Add new source: jsmon for subdomain discovery #1617
Conversation
WalkthroughAdded a new subscraping source package Changes
Sequence Diagram(s)sequenceDiagram
participant Runner
participant jsmon.Source
participant JSMonAPI
Runner->>jsmon.Source: Run(ctx, domain, session)
jsmon.Source->>jsmon.Source: Check API keys
alt No keys
jsmon.Source-->>Runner: Emit skipped, close channel
else Keys present
jsmon.Source->>JSMonAPI: POST /api/v2/subfinderScan2 (domain) with X-Jsmon-Key
JSMonAPI-->>jsmon.Source: 200 + JSON / non-200 or network error
alt 200 + valid JSON
jsmon.Source-->>Runner: Stream Subdomain results
else Error or invalid response
jsmon.Source-->>Runner: Emit Error result
end
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
Poem
Tip 🔌 Remote MCP (Model Context Protocol) integration is now available!Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats. ✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
CodeRabbit Configuration File (
|
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
v2/pkg/subscraping/sources/jsmon/jsmon.go (1)
15-19: Remove unused struct definition.The
subdomainsResponsestruct is defined but never used in the code. This appears to be leftover code that should be removed to avoid confusion.-type subdomainsResponse struct { - Subdomains []string `json:"subdomains"` - Status string `json:"status"` - Message string `json:"message"` -}
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
v2/pkg/passive/sources.go(2 hunks)v2/pkg/runner/options.go(1 hunks)v2/pkg/subscraping/sources/jsmon/jsmon.go(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
v2/pkg/passive/sources.go (1)
Learnt from: x-stp
PR: projectdiscovery/subfinder#0
File: :0-0
Timestamp: 2025-06-24T16:03:30.467Z
Learning: When fixing HTTP response handling bugs in subfinder sources, the correct pattern is to use `defer session.DiscardHTTPResponse(resp)` after successful requests to ensure the response body remains open for reading, and call `session.DiscardHTTPResponse(resp)` immediately in error cases.
🧬 Code Graph Analysis (2)
v2/pkg/passive/sources.go (2)
v2/pkg/subscraping/sources/jsmon/jsmon.go (1)
Source(21-27)v2/pkg/subscraping/types.go (1)
Source(37-62)
v2/pkg/subscraping/sources/jsmon/jsmon.go (1)
v2/pkg/subscraping/types.go (2)
Session(71-78)Statistics(29-34)
🔇 Additional comments (6)
v2/pkg/subscraping/sources/jsmon/jsmon.go (3)
61-71: LGTM: URL construction and request preparation.The URL construction and request body preparation logic is well-implemented. The use of query parameters for workspace ID and proper JSON formatting for the request body follows good practices.
122-149: LGTM: Interface method implementations.All required interface methods are properly implemented with appropriate return values. The source correctly indicates it needs API keys, supports recursive enumeration, and is not enabled by default.
72-79: JSMon HTTP response cleanup is correctly implemented
Thejsmonsource callssession.DiscardHTTPResponse(resp)immediately on every error path and once on the success path before processing results, matching the pattern used throughout other sources. No changes needed.v2/pkg/runner/options.go (1)
258-258: LGTM: Netlas rate limit removal.The netlas rate limit has been properly commented out with a clear explanation. This change aligns with the removal of the netlas source from the passive sources list.
v2/pkg/passive/sources.go (2)
37-37: LGTM: Jsmon import addition.The import for the jsmon source is correctly placed in alphabetical order within the existing imports.
69-69: LGTM: Jsmon source registration.The jsmon source is properly registered in the AllSources array following the established pattern. The placement maintains the existing structure of the array.
|
Please take a look at the following
|
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (1)
v2/pkg/subscraping/sources/jsmon/jsmon.go (1)
55-64: Remove debug print, validate API key parts, and improve error signaling.The debug Printf should be removed. Also validate both parts of the key to avoid empty wkspID/auth token and emit a clear error.
Apply this diff:
- // Find the last two colons to split properly + // Split on the last colon to allow colons in the auth token: <authToken>:<wkspID> lastColonIndex := strings.LastIndex(apiKeyString, ":") if lastColonIndex == -1 { - fmt.Printf("[DEBUG] No colons found in API key\n") s.skipped = true + results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf("invalid jsmon api key format: expected <authToken>:<wkspID>")} return } - authToken = apiKeyString[:lastColonIndex] - wkspId = apiKeyString[lastColonIndex+1:] + authToken = apiKeyString[:lastColonIndex] + wkspId = apiKeyString[lastColonIndex+1:] + if authToken == "" || wkspId == "" { + s.skipped = true + results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf("invalid jsmon api key: empty auth token or wkspID")} + return + }
🧹 Nitpick comments (8)
v2/pkg/subscraping/sources/jsmon/jsmon.go (8)
76-83: Adopt the standard response handling pattern: defer DiscardHTTPResponse on success, guard in error paths.Align with subfinder sources best-practice: defer Discard after a successful request, and call it immediately (nil-guarded) on errors. This avoids double-closing and keeps the body available for decoding.
Apply this diff:
- resp, err := session.Post(ctx, subfinderScanURL, "", headers, bytes.NewReader([]byte(requestBody))) - if err != nil { - // fmt.Printf("[DEBUG] Request error: %v\n", err) - results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err} - s.errors++ - session.DiscardHTTPResponse(resp) - return - } + resp, err := session.Post(ctx, subfinderScanURL, "", headers, bytes.NewReader([]byte(requestBody))) + if err != nil { + if resp != nil { + session.DiscardHTTPResponse(resp) + } + results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err} + s.errors++ + return + } + defer session.DiscardHTTPResponse(resp) @@ - if resp.StatusCode != 200 { + if resp.StatusCode != 200 { // Read response body for error details body, _ := io.ReadAll(resp.Body) // fmt.Printf("[DEBUG] Error response body: %s\n", string(body)) results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf("subfinderScan API returned status %d: %s", resp.StatusCode, string(body))} s.errors++ - session.DiscardHTTPResponse(resp) return } @@ err = jsoniter.NewDecoder(resp.Body).Decode(&response) if err != nil { results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err} s.errors++ - session.DiscardHTTPResponse(resp) return } - - session.DiscardHTTPResponse(resp)Also applies to: 87-95, 97-106
69-76: Build JSON body with proper marshaling and add Accept header.Avoid manual string construction; marshal a struct/map. Also advertise JSON via Accept.
Apply this diff:
- requestBody := fmt.Sprintf(`{"domain":"%s"}`, domain) - - headers := map[string]string{ + requestPayload := map[string]string{"domain": domain} + requestBody, err := jsoniter.Marshal(requestPayload) + if err != nil { + results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf("marshal request body: %w", err)} + s.errors++ + return + } + + headers := map[string]string{ "X-Jsmon-Key": authToken, "Content-Type": "application/json", + "Accept": "application/json", } - resp, err := session.Post(ctx, subfinderScanURL, "", headers, bytes.NewReader([]byte(requestBody))) + resp, err := session.Post(ctx, subfinderScanURL, "", headers, bytes.NewReader(requestBody))Also applies to: 71-74
87-95: Consider using http.StatusOK and handling other “empty-success” statuses if applicable.If jsmon uses 204/202 to signal empty results or async accepted, you may want to treat them accordingly in-source per project convention.
Run-time check suggestion (requires adding net/http import):
- if resp.StatusCode != 200 { + if resp.StatusCode != http.StatusOK {Would you confirm from jsmon docs whether 204 (no content) or 202 (accepted) can be returned for successful-but-empty or async flows? If yes, we can adapt handling locally in this source.
108-111: Respect context cancellation while streaming results.Ensure we stop emitting if the context is canceled mid-stream.
Apply this diff:
- for _, subdomain := range response.Subdomains { - results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain} - s.results++ - } + for _, subdomain := range response.Subdomains { + select { + case <-ctx.Done(): + return + case results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}: + s.results++ + } + }
16-18: Align identifier casing for initialisms (URL, ID).Follow Go naming conventions: URL/ID should be capitalized within identifiers.
Apply this diff:
-const ( - baseUrl = "https://api.jsmon.sh" -) +const ( + baseURL = "https://api.jsmon.sh" +) @@ - var authToken, wkspId string + var authToken, wkspID string @@ - wkspId = apiKeyString[lastColonIndex+1:] + wkspID = apiKeyString[lastColonIndex+1:] @@ - subfinderScanURL := fmt.Sprintf("%s/api/v2/subfinderScan2?wkspId=%s", baseUrl, wkspId) + subfinderScanURL := fmt.Sprintf("%s/api/v2/subfinderScan2?wkspId=%s", baseURL, wkspID)Also applies to: 51-64, 67-68
4-14: Use http.Status constants instead of magic numbers.*Import net/http and replace 200 with http.StatusOK for readability.
Apply this diff:
import ( "bytes" "context" "fmt" "io" + "net/http" "strings" "time" @@ - if resp.StatusCode != 200 { + if resp.StatusCode != http.StatusOK {Also applies to: 87-87
78-86: Remove commented-out debug logs.Leftover commented debug lines add noise.
Apply this diff:
- if err != nil { - // fmt.Printf("[DEBUG] Request error: %v\n", err) + if err != nil { @@ - // fmt.Printf("[DEBUG] Response status: %d\n", resp.StatusCode) @@ - // fmt.Printf("[DEBUG] Error response body: %s\n", string(body))Also applies to: 90-90
1-3: Outstanding PR tasks: resolve conflicts and update README.Per maintainer comment on the PR, please resolve merge conflicts and add README docs for jsmon usage (key format, rate limits, example).
I can draft the README section (including API key format : and configuration instructions). Want me to propose that?
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
v2/pkg/subscraping/sources/jsmon/jsmon.go(1 hunks)
🧰 Additional context used
🧠 Learnings (2)
📚 Learning: 2025-06-24T16:03:30.467Z
Learnt from: x-stp
PR: projectdiscovery/subfinder#0
File: :0-0
Timestamp: 2025-06-24T16:03:30.467Z
Learning: When fixing HTTP response handling bugs in subfinder sources, the correct pattern is to use `defer session.DiscardHTTPResponse(resp)` after successful requests to ensure the response body remains open for reading, and call `session.DiscardHTTPResponse(resp)` immediately in error cases.
Applied to files:
v2/pkg/subscraping/sources/jsmon/jsmon.go
📚 Learning: 2025-07-17T12:07:51.521Z
Learnt from: 0x4500
PR: projectdiscovery/subfinder#1612
File: pkg/subscraping/agent.go:0-0
Timestamp: 2025-07-17T12:07:51.521Z
Learning: In subfinder subscraping sources, when a source needs to handle specific HTTP status codes differently (like treating 204 as success), the check should be implemented within the individual source's code rather than modifying the global httpRequestWrapper in agent.go. This keeps the special handling localized and avoids affecting other sources.
Applied to files:
v2/pkg/subscraping/sources/jsmon/jsmon.go
🧬 Code Graph Analysis (1)
v2/pkg/subscraping/sources/jsmon/jsmon.go (1)
v2/pkg/subscraping/types.go (2)
Session(71-78)Statistics(29-34)
🔇 Additional comments (2)
v2/pkg/subscraping/sources/jsmon/jsmon.go (2)
34-38: Potential data race if Source is reused concurrently.s.errors/results/timeTaken are mutated per Run. If the same Source instance runs concurrently for multiple domains, it can race. If the runner instantiates per run, you’re fine.
Can you confirm the lifecycle in this codebase (one Source instance per run)? If not, we should guard these fields or avoid storing them on the receiver.
122-124: Confirm IsDefault() for a key-required source.Returning true includes jsmon in the default set; without keys it’s skipped (as you handle), which is acceptable. Just confirm this matches repository convention for keyed sources.
Using Jsmon as a Data Source in SubfinderJsmon is now available as a subdomain data source in Subfinder. To use it, you need a Jsmon API key and a workspace ID. 1. Clone and build SubfinderGet your Jsmon API Key and Workspace IDConfigure Subfinder for JsmonThe Jsmon configuration format is: Run Subfinder with JsmonNow you can run Subfinder using Jsmon as a source: |
|
@dogancanbakir, Please check the instructions for using the jsmon as a data source. About the conflicts, since the file is huge, I am not able to resolve the conflicts from the github directly. Can you please resolve the conflicts from your end. |
|
Suppressed by #1637 |
|
@nandini-56 Please take a look at this #1637 |
This PR adds a new passive source named 'jsmon' for discovering subdomains.
Changes made:
Tested and verified output using the 'subfinder' tool. This source contributes to better subdomain coverage using jsmon data.
Looking forward to feedback or suggestions from the maintainers.
Summary by CodeRabbit
New Features
Chores