add windvane #1657

dogancanbakir · 2025-10-16T11:57:52Z

Summary by CodeRabbit

New Features
- Added Windvane as a new subdomain discovery source — supports API keys, optional recursive lookups, and reporting of run statistics.
Tests
- Updated source tests to include Windvane among available and default sources.

coderabbitai · 2025-10-16T11:58:06Z

Walkthrough

A new Windvane subscraping source was added and registered: implementation of the Source type with API communication, pagination, and result emission, and the passive sources registry and tests updated to include Windvane.

Changes

Cohort / File(s)	Summary
Passive sources registry `pkg/passive/sources.go`	Adds `windvane` import and appends `&windvane.Source{}` to the `AllSources` slice for passive source enumeration.
Test data `pkg/passive/sources_test.go`	Updates `expectedAllSources` and `expectedDefaultSources` to include the Windvane source in test assertions.
Windvane source implementation `pkg/subscraping/sources/windvane/windvane.go`	Adds new `Source` type implementing the subscraping interface with methods `Run`, `Name`, `IsDefault`, `HasRecursiveSupport`, `NeedsKey`, `AddApiKeys`, and `Statistics`. Implements API POST requests, response decoding, pagination, event emission, and error/time tracking using `subscraping.Session`.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant Orchestrator as Orchestrator
  participant Source as windvane.Source
  participant Session as sub.scraping.Session
  participant API as Windvane API

  Note over Orchestrator,Source: Orchestrator triggers source.Run(domain)
  Orchestrator->>Source: Run(ctx, domain, session)
  loop for each page
    Source->>Session: POST /search (payload with domain, page, apiKey)
    Session->>API: HTTP POST request
    API-->>Session: JSON response (results + pagination)
    Session-->>Source: decoded response
    alt has results
      Source-->>Orchestrator: emit subscraping.Result (subdomain events)
    end
    alt more pages
      Source->>Source: increment page and continue
    else done
      Source-->>Orchestrator: close result channel
    end
  end

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 I found a new breeze called Windvane today,
It hops through pages, fetching names in play,
Keys in my pouch and code in my paw,
I scatter subdomains — a curious awe! 🌬️

Pre-merge checks and finishing touches

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check	✅ Passed	The PR title "add windvane" clearly and directly describes the main objective of the pull request, which is to integrate the windvane source into the project. The title is concise, specific enough to understand the primary change, and accurately reflects the modifications made across all three files: adding windvane to the passive sources registry, updating corresponding tests, and implementing the new windvane data source. The title avoids vague terminology and provides meaningful information about the changeset.
Linked Issues Check	✅ Passed	The linked issue #1639 requires integrating a new data source named "windvane" into the project using the Windvane API. The PR satisfies this requirement by introducing a complete windvane source implementation that follows the subscraping interface, adding it to the passive sources registry in `pkg/passive/sources.go`, and updating the corresponding tests in `pkg/passive/sources_test.go`. The new `windvane.go` file implements all required interface methods including Run, Name, IsDefault, HasRecursiveSupport, NeedsKey, AddApiKeys, and Statistics, along with API key management and HTTP request handling for fetching subdomains with pagination support.
Out of Scope Changes Check	✅ Passed	All changes in this PR are directly related to the objective of adding the windvane source. The modifications include adding windvane to the passive sources registry, updating tests to include the new source, and implementing the complete windvane data source. No unrelated changes to other functionality, refactoring of unrelated code, or modifications outside the scope of integrating windvane were introduced.
Docstring Coverage	✅ Passed	No functions found in the changes. Docstring coverage check skipped.

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch 1639_add_windvane

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 8117b1a and 26a221f.

📒 Files selected for processing (1)

pkg/subscraping/sources/windvane/windvane.go (1 hunks)

🧰 Additional context used

🧠 Learnings (1)

📚 Learning: 2025-06-24T16:03:30.467Z

Learnt from: x-stp
PR: projectdiscovery/subfinder#0
File: :0-0
Timestamp: 2025-06-24T16:03:30.467Z
Learning: When fixing HTTP response handling bugs in subfinder sources, the correct pattern is to use `defer session.DiscardHTTPResponse(resp)` after successful requests to ensure the response body remains open for reading, and call `session.DiscardHTTPResponse(resp)` immediately in error cases.

Applied to files:

pkg/subscraping/sources/windvane/windvane.go

🧬 Code graph analysis (1)

pkg/subscraping/sources/windvane/windvane.go (2)

pkg/subscraping/types.go (2)

Session (71-78)

Statistics (29-34)

pkg/subscraping/utils.go (1)

PickRandom (12-20)

🔇 Additional comments (7)

pkg/subscraping/sources/windvane/windvane.go (7)

1-13: Package and imports look good.

All necessary imports are present and appropriately used for JSON handling, HTTP operations, and pagination logic.

15-34: Response type definitions are appropriate.

The struct definitions correctly map to the Windvane API response format. The string types for pagination fields in pageInfo are handled via conversion in the pagination logic.

36-42: Source struct implementation is clean.

The struct properly encapsulates state for API keys, timing, and statistics tracking.

44-62: Initialization and API key handling are correct.

The method properly initializes the results channel, resets counters, selects a random API key, and configures headers for the Windvane API.

91-94: Result emission logic is correct.

Subdomains are properly emitted as results with appropriate type and value, and the results counter is incremented.

106-111: Pagination boundary check is correct.

The calculation (page-1)*recordsPerPage >= totalRecords properly determines when all pages have been fetched, and the page increment ensures forward progress.

118-145: Interface implementation methods are correct.

All required interface methods are properly implemented with appropriate return values for the Windvane source characteristics (default enabled, requires API key, no recursive support).

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 3

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c47f852 and 8117b1a.

📒 Files selected for processing (3)

pkg/passive/sources.go (2 hunks)
pkg/passive/sources_test.go (2 hunks)
pkg/subscraping/sources/windvane/windvane.go (1 hunks)

🧰 Additional context used

🧬 Code graph analysis (2)

pkg/passive/sources.go (2)

pkg/subscraping/sources/windvane/windvane.go (1)

Source (37-43)

pkg/subscraping/types.go (1)

Source (37-62)

pkg/subscraping/sources/windvane/windvane.go (2)

pkg/subscraping/types.go (2)

Session (71-78)

Statistics (29-34)

pkg/subscraping/utils.go (1)

PickRandom (12-20)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)

GitHub Check: Test Builds (macOS-latest)
GitHub Check: Test Builds (windows-latest)
GitHub Check: Test Builds (ubuntu-latest)
GitHub Check: Analyze (go)

🔇 Additional comments (3)

pkg/passive/sources_test.go (1)

51-51: LGTM!

The windvane source is correctly added to both test arrays, maintaining alphabetical order and aligning with the source's IsDefault() = true implementation.

Also applies to: 89-89

pkg/passive/sources.go (1)

54-54: LGTM!

The windvane source is properly imported and registered, following the established patterns for other sources.

Also applies to: 99-99

pkg/subscraping/sources/windvane/windvane.go (1)

119-146: LGTM!

All interface methods correctly implement the subscraping.Source interface with appropriate return values that align with the source's registration and test expectations.

pkg/subscraping/sources/windvane/windvane.go

Mzack9999 · 2025-10-19T11:28:19Z

@coderabbitai resolve

coderabbitai · 2025-10-19T11:28:31Z

✅ Actions performed

Comments resolved.

Mzack9999

lgtm!

add windvane

8117b1a

dogancanbakir requested a review from Mzack9999 October 16, 2025 11:57

dogancanbakir self-assigned this Oct 16, 2025

coderabbitai bot reviewed Oct 16, 2025

View reviewed changes

pkg/subscraping/sources/windvane/windvane.go Outdated Show resolved Hide resolved

pkg/subscraping/sources/windvane/windvane.go Show resolved Hide resolved

pkg/subscraping/sources/windvane/windvane.go Show resolved Hide resolved

using json marshal

26a221f

Mzack9999 approved these changes Oct 19, 2025

View reviewed changes

dogancanbakir merged commit a8e87b7 into dev Oct 20, 2025
10 checks passed

dogancanbakir deleted the 1639_add_windvane branch October 20, 2025 07:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

add windvane #1657

add windvane #1657

Uh oh!

dogancanbakir commented Oct 16, 2025 •

edited by coderabbitai bot

Loading

Uh oh!

coderabbitai bot commented Oct 16, 2025 •

edited

Loading

Uh oh!

coderabbitai bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Mzack9999 commented Oct 19, 2025

Uh oh!

coderabbitai bot commented Oct 19, 2025

Uh oh!

Mzack9999 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

add windvane #1657

add windvane #1657

Uh oh!

Conversation

dogancanbakir commented Oct 16, 2025 • edited by coderabbitai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai bot commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Poem

Pre-merge checks and finishing touches

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Mzack9999 commented Oct 19, 2025

Uh oh!

coderabbitai bot commented Oct 19, 2025

Uh oh!

Mzack9999 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dogancanbakir commented Oct 16, 2025 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Oct 16, 2025 •

edited

Loading