add apikey support for hackertarget #1622
Conversation
WalkthroughThe changes introduce optional API key support for the HackerTarget source, enabling random API key selection and skipping execution if no key is provided. The configuration logic for adding API keys to sources is also simplified, now always attempting to add keys if any are present, regardless of whether the source explicitly requires them. Changes
Sequence Diagram(s)sequenceDiagram
participant Config as Config Loader
participant Source as HackerTarget Source
participant User as User
Config->>Source: AddApiKeys(apiKeys)
User->>Source: Run(domain, session)
alt apiKeys present
Source->>Source: PickRandom(apiKey)
Source->>Source: Construct URL with &apikey=<key>
Source->>Source: HTTP GET to HackerTarget
Source-->>User: Return results
else No apiKeys
Source->>Source: Set skipped = true
Source-->>User: Return (no results)
end
Estimated code review effort2 (~15 minutes) Possibly related PRs
Suggested reviewers
Poem
✨ Finishing Touches
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
pkg/runner/config.go(1 hunks)pkg/subscraping/sources/hackertarget/hackertarget.go(3 hunks)
🧠 Learnings (2)
pkg/runner/config.go (1)
Learnt from: 0x4500
PR: #1612
File: pkg/subscraping/agent.go:0-0
Timestamp: 2025-07-17T12:07:51.521Z
Learning: In subfinder subscraping sources, when a source needs to handle specific HTTP status codes differently (like treating 204 as success), the check should be implemented within the individual source's code rather than modifying the global httpRequestWrapper in agent.go. This keeps the special handling localized and avoids affecting other sources.
pkg/subscraping/sources/hackertarget/hackertarget.go (3)
Learnt from: 0x4500
PR: #1612
File: pkg/subscraping/agent.go:0-0
Timestamp: 2025-07-17T12:07:51.521Z
Learning: In subfinder subscraping sources, when a source needs to handle specific HTTP status codes differently (like treating 204 as success), the check should be implemented within the individual source's code rather than modifying the global httpRequestWrapper in agent.go. This keeps the special handling localized and avoids affecting other sources.
Learnt from: x-stp
PR: projectdiscovery/subfinder#0
File: :0-0
Timestamp: 2025-06-24T16:03:30.467Z
Learning: When fixing HTTP response handling bugs in subfinder sources, the correct pattern is to use defer session.DiscardHTTPResponse(resp) after successful requests to ensure the response body remains open for reading, and call session.DiscardHTTPResponse(resp) immediately in error cases.
Learnt from: x-stp
PR: #1608
File: v2/pkg/subscraping/sources/shrewdeye/shrewdeye.go:32-38
Timestamp: 2025-06-20T19:02:59.053Z
Learning: The DiscardHTTPResponse method in subfinder's Session already includes a built-in nil check for the response parameter, so it's safe to call with a potentially nil http.Response without additional nil checking.
🧰 Additional context used
🧠 Learnings (2)
pkg/runner/config.go (1)
Learnt from: 0x4500
PR: #1612
File: pkg/subscraping/agent.go:0-0
Timestamp: 2025-07-17T12:07:51.521Z
Learning: In subfinder subscraping sources, when a source needs to handle specific HTTP status codes differently (like treating 204 as success), the check should be implemented within the individual source's code rather than modifying the global httpRequestWrapper in agent.go. This keeps the special handling localized and avoids affecting other sources.
pkg/subscraping/sources/hackertarget/hackertarget.go (3)
Learnt from: 0x4500
PR: #1612
File: pkg/subscraping/agent.go:0-0
Timestamp: 2025-07-17T12:07:51.521Z
Learning: In subfinder subscraping sources, when a source needs to handle specific HTTP status codes differently (like treating 204 as success), the check should be implemented within the individual source's code rather than modifying the global httpRequestWrapper in agent.go. This keeps the special handling localized and avoids affecting other sources.
Learnt from: x-stp
PR: projectdiscovery/subfinder#0
File: :0-0
Timestamp: 2025-06-24T16:03:30.467Z
Learning: When fixing HTTP response handling bugs in subfinder sources, the correct pattern is to use defer session.DiscardHTTPResponse(resp) after successful requests to ensure the response body remains open for reading, and call session.DiscardHTTPResponse(resp) immediately in error cases.
Learnt from: x-stp
PR: #1608
File: v2/pkg/subscraping/sources/shrewdeye/shrewdeye.go:32-38
Timestamp: 2025-06-20T19:02:59.053Z
Learning: The DiscardHTTPResponse method in subfinder's Session already includes a built-in nil check for the response parameter, so it's safe to call with a potentially nil http.Response without additional nil checking.
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (5)
- GitHub Check: Test Builds (windows-latest)
- GitHub Check: Test Builds (ubuntu-latest)
- GitHub Check: Test Builds (macOS-13)
- GitHub Check: release-test
- GitHub Check: Analyze (go)
🔇 Additional comments (5)
pkg/runner/config.go (1)
49-49: LGTM! Simplified logic aligns with optional API key pattern.The removal of the
source.NeedsKey()check enables sources to receive API keys even when they don't strictly require them, allowing for optional API key usage. The nil check removal is safe sincelen()on a nil slice returns 0.pkg/subscraping/sources/hackertarget/hackertarget.go (4)
15-15: LGTM! Struct fields properly support optional API key functionality.The addition of
apiKeysandskippedfields enables tracking of API key state and execution status as intended.Also applies to: 19-19
43-49: LGTM! HTTP request handling follows established patterns.The HTTP request implementation correctly uses
session.SimpleGet()and proper error handling withsession.DiscardHTTPResponse()as per subfinder conventions.
92-95: LGTM! API key storage implementation is correct.The
AddApiKeysmethod now properly stores the provided keys, and the TODO comment appropriately highlights the broader refactoring needs for API key management.
102-102: LGTM! Statistics correctly report skip status.The addition of the
Skippedfield to statistics provides proper visibility into when the source was skipped due to missing API keys.
| htSearchUrl := fmt.Sprintf("https://api.hackertarget.com/hostsearch/?q=%s", domain) | ||
| randomApiKey := subscraping.PickRandom(s.apiKeys, s.Name()) | ||
| if randomApiKey == "" { | ||
| s.skipped = true | ||
| return | ||
| } | ||
|
|
||
| htSearchUrl = fmt.Sprintf("%s&apikey=%s", htSearchUrl, randomApiKey) | ||
|
|
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
Verify that HackerTarget API requires authentication.
The implementation skips execution entirely when no API key is available, but this may be overly restrictive if HackerTarget's API supports unauthenticated requests with rate limits.
Please verify if HackerTarget's API works without authentication:
🌐 Web query:
Does HackerTarget's hostsearch API work without an API key, and what are the rate limits for authenticated vs unauthenticated requests?
💡 Result:
HackerTarget's hostsearch API is accessible without an API key, allowing users to perform queries without authentication. However, unauthenticated users are subject to certain rate limits to ensure fair usage and maintain service stability.
Rate Limits for Unauthenticated Users:
-
Daily Limit: Unauthenticated users are permitted up to 50 API calls per day from a single IP address. (hackertarget.com)
-
Per-Second Limit: The API enforces a rate limit of 2 requests per second from a single IP address. Exceeding this limit will result in an HTTP 429 response code. (hackertarget.com)
Rate Limits for Authenticated Users:
Authenticated users, who have a HackerTarget membership and possess an API key, enjoy enhanced access:
-
Unlimited Results: Authenticated users can retrieve unlimited results, whereas free queries are limited to 500 results. (hackertarget.com)
-
Generous Daily Quota: The daily quota for authenticated users is based on their membership plan, offering a higher allowance compared to unauthenticated users. (hackertarget.com)
-
Additional Features: Membership provides access to advanced tools like Attack Surface Mapping with Domain Profiler and full access to 28 vulnerability scanners and tools. (hackertarget.com)
To access the API as an authenticated user, you need to sign up for a HackerTarget membership. Once registered, you can find your API key in the user dashboard. This key can be included in your API requests either as a query parameter (&apikey=) or as a custom header (X-API-Key). (hackertarget.com)
By adhering to these rate limits and utilizing the appropriate authentication, you can effectively use HackerTarget's hostsearch API for your network security assessments.
Citations:
- 1: https://hackertarget.com/ip-tools/?utm_source=openai
- 2: https://hackertarget.com/ip-tools/?utm_source=openai
- 3: https://hackertarget.com/ip-tools/?utm_source=openai
- 4: https://hackertarget.com/ip-tools/?utm_source=openai
- 5: https://hackertarget.com/ip-tools/?utm_source=openai
- 6: https://hackertarget.com/ip-tools/?utm_source=openai
Allow Unauthenticated HackerTarget API Requests
HackerTarget’s hostsearch endpoint supports unauthenticated queries (up to 50 calls/day and 2 calls/sec per IP). Skipping entirely when no API key is present is overly restrictive—let’s fall back to an unauthenticated request instead.
Locations to update:
- pkg/subscraping/sources/hackertarget/hackertarget.go: lines 34–42
Suggested diff:
- randomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())
- if randomApiKey == "" {
- s.skipped = true
- return
- }
-
- htSearchUrl = fmt.Sprintf("%s&apikey=%s", htSearchUrl, randomApiKey)
+ randomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())
+ if randomApiKey != "" {
+ // Authenticated request: unlimited results per plan
+ htSearchUrl = fmt.Sprintf("%s&apikey=%s", htSearchUrl, randomApiKey)
+ } else {
+ // Unauthenticated fallback: limited to 50 calls/day and 2 calls/sec
+ // Proceed without &apikey parameter
+ }• Remove s.skipped = true and return to allow unauthenticated queries
• Optionally, add logging to warn when falling back to unauthenticated mode
• Be mindful of rate limits (50/day, 2/sec) to avoid HTTP 429 errors
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| htSearchUrl := fmt.Sprintf("https://api.hackertarget.com/hostsearch/?q=%s", domain) | |
| randomApiKey := subscraping.PickRandom(s.apiKeys, s.Name()) | |
| if randomApiKey == "" { | |
| s.skipped = true | |
| return | |
| } | |
| htSearchUrl = fmt.Sprintf("%s&apikey=%s", htSearchUrl, randomApiKey) | |
| htSearchUrl := fmt.Sprintf("https://api.hackertarget.com/hostsearch/?q=%s", domain) | |
| randomApiKey := subscraping.PickRandom(s.apiKeys, s.Name()) | |
| if randomApiKey != "" { | |
| // Authenticated request: unlimited results per plan | |
| htSearchUrl = fmt.Sprintf("%s&apikey=%s", htSearchUrl, randomApiKey) | |
| } else { | |
| // Unauthenticated fallback: limited to 50 calls/day and 2 calls/sec | |
| // Proceed without &apikey parameter | |
| } |
🤖 Prompt for AI Agents
In pkg/subscraping/sources/hackertarget/hackertarget.go around lines 34 to 42,
the current code skips the request entirely if no API key is available, which is
too restrictive since HackerTarget allows unauthenticated queries with limits.
Remove the lines setting s.skipped to true and the return statement to allow the
request to proceed without an API key. Optionally, add a log warning indicating
fallback to unauthenticated mode. Ensure the URL is constructed without the
apikey parameter when no key is present, and consider implementing rate limiting
to respect the 50 calls/day and 2 calls/sec limits.
closes #1576
Summary by CodeRabbit
New Features
Bug Fixes