这是indexloc提供的服务,不要输入任何密码
Skip to content

Conversation

@dogancanbakir
Copy link
Member

@dogancanbakir dogancanbakir commented Aug 26, 2025

closes #1636

Summary by CodeRabbit

  • New Features
    • Added jsmon as a new passive source for subdomain discovery, enabled by default.
    • Requires an API key and supports multiple keys; one is chosen automatically during scans.
    • Improves coverage and breadth of results; provides run statistics.
  • Tests
    • Updated test expectations to include jsmon in the default and all sources sets.

@coderabbitai
Copy link

coderabbitai bot commented Aug 26, 2025

Walkthrough

Adds a new passive subscraping source "jsmon": registers it in AllSources, implements its Run flow to call the jsmon API with token/workspace key format, emits discovered subdomains, exposes statistics, and updates tests to expect "jsmon" in all/default sources.

Changes

Cohort / File(s) Summary
Register jsmon source
pkg/passive/sources.go
Imports jsmon and adds &jsmon.Source{} to AllSources.
Tests updated for jsmon
pkg/passive/sources_test.go
Adds "jsmon" to expectedAllSources and expectedDefaultSources.
New jsmon implementation
pkg/subscraping/sources/jsmon/jsmon.go
Introduces jsmon.Source with API key handling, Run invoking jsmon API, result emission, error handling, stats, and metadata methods (Name, IsDefault, HasRecursiveSupport, NeedsKey, AddApiKeys, Statistics).

Sequence Diagram(s)

sequenceDiagram
    autonumber
    participant User
    participant Subfinder as Passive Engine
    participant JSMon as jsmon.Source
    participant API as jsmon API

    User->>Subfinder: subdomain enumeration (domain)
    Subfinder->>JSMon: Run(ctx, domain, session)
    rect rgba(200,235,255,0.3)
      note right of JSMon: Select API key (token:wkspId)<br/>Set headers and body
      JSMon->>API: POST /api/v2/subfinderScan2?wkspId=...<br/>X-Jsmon-Key: token<br/>{ "domain": "<domain>" }
      API-->>JSMon: 200 OK + { subdomains: [...] } or error
    end
    alt 200 OK
      loop for each subdomain
        JSMon-->>Subfinder: Result{Type: Subdomain, Value}
      end
    else non-200 / network error
      JSMon-->>Subfinder: Result{Type: Error, Info}
    end
    JSMon-->>Subfinder: Statistics{Results, Errors, TimeTaken, Skipped}
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Assessment against linked issues

Objective Addressed Explanation
Add jsmon source to subscraping/passive system (#1636)

Assessment against linked issues: Out-of-scope changes

None found.

Suggested reviewers

  • ehsandeep

Poem

Thump-thump go my paws on the keys tonight,
New trails in the grass where subdomains alight—
JSMon’s whisper, a carrot-bright clue,
I nibble the bytes, find hosts anew.
Ears up, tails high, results in a run,
Hop-hop, ship it—this patch is done! 🥕🐇

Tip

🔌 Remote MCP (Model Context Protocol) integration is now available!

Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats.

✨ Finishing Touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch add_jsmon

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

  • Visit our Status Page to check the current availability of CodeRabbit.
  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (8)
pkg/passive/sources.go (1)

106-107: Registered &jsmon.Source{} — consider adding a user-facing warning for API key format

Since jsmon expects JSMON_API_KEY in "token:workspaceID" form, surface a friendly warning like other sources to reduce misconfiguration churn.

You can set a warning once during package init:

 var sourceWarnings = mapsutil.NewSyncLockMap[string, string](
   mapsutil.WithMap(mapsutil.Map[string, string]{}))
 
+func init() {
+  // jsmon requires "token:workspaceID" format
+  sourceWarnings.Set("jsmon", "Set JSMON_API_KEY as <TOKEN>:<WORKSPACE_ID> (e.g., abc123:ws_456).")
+}

Note: multiple init() functions are allowed in the same package.

pkg/subscraping/sources/jsmon/jsmon.go (7)

35-39: Reset skipped flag per run to avoid stale statistics

s.skipped is only ever set to true; if a previous run skipped, subsequent successful runs will still report Skipped=true.

 func (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {
   results := make(chan subscraping.Result)
   s.errors = 0
   s.results = 0
+  s.skipped = false

52-60: API key parsing: use SplitN and trim; adopt Go ID casing

Tokens can unexpectedly contain a colon, and whitespace can sneak in. Also prefer wkspID for Go style.

-    randomApiInfo := strings.Split(randomApiKey, ":")
-    if len(randomApiInfo) != 2 {
+    parts := strings.SplitN(strings.TrimSpace(randomApiKey), ":", 2)
+    if len(parts) != 2 {
       s.skipped = true
       return
     }
-
-    authToken := randomApiInfo[0]
-    wkspId := randomApiInfo[1]
+    authToken := strings.TrimSpace(parts[0])
+    wkspID := strings.TrimSpace(parts[1])

61-67: Build JSON payload via marshaling, and prefer baseURL/wkspID naming

String interpolation risks malformed JSON if domain ever contains characters needing escaping; marshaling is safer. Also normalize URL/ID names.

-    subfinderScanURL := fmt.Sprintf("%s/api/v2/subfinderScan2?wkspId=%s", baseUrl, wkspId)
-    requestBody := fmt.Sprintf(`{"domain":"%s"}`, domain)
+    subfinderScanURL := fmt.Sprintf("%s/api/v2/subfinderScan2?wkspId=%s", baseURL, wkspID)
+    payload, _ := jsoniter.Marshal(struct {
+      Domain string `json:"domain"`
+    }{Domain: domain})
     headers := map[string]string{
       "X-Jsmon-Key":  authToken,
       "Content-Type": "application/json",
     }
 
-    resp, err := session.Post(ctx, subfinderScanURL, "", headers, bytes.NewReader([]byte(requestBody)))
+    resp, err := session.Post(ctx, subfinderScanURL, "", headers, bytes.NewReader(payload))

And rename the constant:

-const (
-  baseUrl = "https://api.jsmon.sh"
-)
+const (
+  baseURL = "https://api.jsmon.sh"
+)

84-97: Optional: handle API-declared error states before emitting subdomains

Response has Status/Message; consider surfacing server-declared errors explicitly instead of silently proceeding.

   var response subdomainsResponse
   if err := jsoniter.NewDecoder(resp.Body).Decode(&response); err != nil {
     results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
     s.errors++
     session.DiscardHTTPResponse(resp)
     return
   }
 
   session.DiscardHTTPResponse(resp)
 
+  if response.Status != "" && !strings.EqualFold(response.Status, "ok") && !strings.EqualFold(response.Status, "success") {
+    results <- subscraping.Result{
+      Source: s.Name(), Type: subscraping.Error,
+      Error: fmt.Errorf("subfinderScan API status=%s message=%s", response.Status, response.Message),
+    }
+    s.errors++
+    return
+  }

1-3: Package comment style nit

Consider an idiomatic package comment.

-// Package jsmon logic
+// Package jsmon implements the Subfinder source backed by the jsmon API.

17-19: Identifier casing nit

Use baseURL per Go style for initialisms.

(See diff proposed in a prior comment.)


108-110: Confirm default inclusion is intended

IsDefault() returns true, so jsmon participates without explicit opt-in. If jsmon commonly requires credentials, default-enabled may surprise users without JSMON_API_KEY configured (it will be marked skipped). If that’s intentional for visibility, ignore this; otherwise consider making it non-default.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

  • MCP integration is disabled by default for public repositories
  • Jira integration is disabled by default for public repositories
  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between b47737d and ca35407.

📒 Files selected for processing (3)
  • pkg/passive/sources.go (2 hunks)
  • pkg/passive/sources_test.go (2 hunks)
  • pkg/subscraping/sources/jsmon/jsmon.go (1 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
pkg/passive/sources.go (1)
pkg/subscraping/sources/jsmon/jsmon.go (1)
  • Source (27-33)
pkg/subscraping/sources/jsmon/jsmon.go (2)
pkg/subscraping/types.go (2)
  • Session (71-78)
  • Statistics (29-34)
pkg/subscraping/utils.go (1)
  • PickRandom (12-20)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (5)
  • GitHub Check: Test Builds (windows-latest)
  • GitHub Check: Test Builds (ubuntu-latest)
  • GitHub Check: Test Builds (macOS-13)
  • GitHub Check: release-test
  • GitHub Check: Analyze (go)
🔇 Additional comments (5)
pkg/passive/sources_test.go (2)

60-61: Added "jsmon" to expectedAllSources — LGTM

Consistent with the new source registration. No issues.


98-99: Added "jsmon" to expectedDefaultSources — LGTM (matches IsDefault=true)

This aligns with jsmon.Source.IsDefault() returning true. expectedDefaultRecursiveSources correctly remains unchanged since HasRecursiveSupport() is false.

pkg/passive/sources.go (1)

38-39: Importing jsmon source — LGTM

Import is accurate and scoped. No side effects.

pkg/subscraping/sources/jsmon/jsmon.go (2)

124-131: Statistics accessor — LGTM

Fields are populated as expected; once the per-run reset of skipped is added, stats will be consistent.


68-75: Possible nil dereference on error path (resp may be nil)

If session.Post fails before creating a response, DiscardHTTPResponse(resp) can panic.

Apply this guard:

-    if err != nil {
-      results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
-      s.errors++
-      session.DiscardHTTPResponse(resp)
-      return
-    }
+    if err != nil {
+      results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
+      s.errors++
+      if resp != nil {
+        session.DiscardHTTPResponse(resp)
+      }
+      return
+    }
⛔ Skipped due to learnings
Learnt from: x-stp
PR: projectdiscovery/subfinder#0
File: :0-0
Timestamp: 2025-06-24T16:03:30.467Z
Learning: When fixing HTTP response handling bugs in subfinder sources, the correct pattern is to use `defer session.DiscardHTTPResponse(resp)` after successful requests to ensure the response body remains open for reading, and call `session.DiscardHTTPResponse(resp)` immediately in error cases.
Learnt from: x-stp
PR: projectdiscovery/subfinder#1608
File: v2/pkg/subscraping/sources/shrewdeye/shrewdeye.go:32-38
Timestamp: 2025-06-20T19:02:59.053Z
Learning: The DiscardHTTPResponse method in subfinder's Session already includes a built-in nil check for the response parameter, so it's safe to call with a potentially nil http.Response without additional nil checking.
Learnt from: x-stp
PR: projectdiscovery/subfinder#0
File: :0-0
Timestamp: 2025-06-20T19:05:25.823Z
Learning: The subfinder session provides a `DiscardHTTPResponse` method that properly handles HTTP response cleanup by draining the body and closing it with error handling. Sources should use this method instead of manually closing response bodies to avoid redundant syscalls.

@dogancanbakir
Copy link
Member Author

After thoroughly testing it, we realized that it can be non-deterministic and occasionally doesn't produce great results in some domains. So, we've decided not to include it for now, but we'll definitely keep it in mind for future considerations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

add jsmon

2 participants