这是indexloc提供的服务，不要输入任何密码
Skip to content

add apikey support for hackertarget #1622

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: dev
Choose a base branch
from
Open

add apikey support for hackertarget #1622

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e3af027
add api key support for hackertarget
dogancanbakir Jul 23, 2025
f786099
remove debug code
dogancanbakir Jul 23, 2025
31d5508
minor
dogancanbakir Jul 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pkg/runner/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ func UnmarshalFrom(file string) error {
for _, source := range passive.AllSources {
sourceName := strings.ToLower(source.Name())
apiKeys := sourceApiKeysMap[sourceName]
if source.NeedsKey() && apiKeys != nil && len(apiKeys) > 0 {
if len(apiKeys) > 0 {
gologger.Debug().Msgf("API key(s) found for %s.", sourceName)
source.AddApiKeys(apiKeys)
}
Expand Down
19 changes: 16 additions & 3 deletions pkg/subscraping/sources/hackertarget/hackertarget.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,11 @@ import (

// Source is the passive scraping agent
type Source struct {
apiKeys []string
timeTaken time.Duration
errors int
results int
skipped bool
}

// Run function returns all subdomains found with the service
Expand All @@ -29,7 +31,16 @@ func (s *Source) Run(ctx context.Context, domain string, session *subscraping.Se
close(results)
}(time.Now())

resp, err := session.SimpleGet(ctx, fmt.Sprintf("https://api.hackertarget.com/hostsearch/?q=%s", domain))
htSearchUrl := fmt.Sprintf("https://api.hackertarget.com/hostsearch/?q=%s", domain)
randomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())
if randomApiKey == "" {
s.skipped = true
return
}

htSearchUrl = fmt.Sprintf("%s&apikey=%s", htSearchUrl, randomApiKey)

Comment on lines +34 to +42
Copy link

@coderabbitai coderabbitai bot Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Verify that HackerTarget API requires authentication.

The implementation skips execution entirely when no API key is available, but this may be overly restrictive if HackerTarget's API supports unauthenticated requests with rate limits.

Please verify if HackerTarget's API works without authentication:

🌐 Web query:

Does HackerTarget's hostsearch API work without an API key, and what are the rate limits for authenticated vs unauthenticated requests?

💡 Result:

HackerTarget's hostsearch API is accessible without an API key, allowing users to perform queries without authentication. However, unauthenticated users are subject to certain rate limits to ensure fair usage and maintain service stability.

Rate Limits for Unauthenticated Users:

  • Daily Limit: Unauthenticated users are permitted up to 50 API calls per day from a single IP address. (hackertarget.com)

  • Per-Second Limit: The API enforces a rate limit of 2 requests per second from a single IP address. Exceeding this limit will result in an HTTP 429 response code. (hackertarget.com)

Rate Limits for Authenticated Users:

Authenticated users, who have a HackerTarget membership and possess an API key, enjoy enhanced access:

  • Unlimited Results: Authenticated users can retrieve unlimited results, whereas free queries are limited to 500 results. (hackertarget.com)

  • Generous Daily Quota: The daily quota for authenticated users is based on their membership plan, offering a higher allowance compared to unauthenticated users. (hackertarget.com)

  • Additional Features: Membership provides access to advanced tools like Attack Surface Mapping with Domain Profiler and full access to 28 vulnerability scanners and tools. (hackertarget.com)

To access the API as an authenticated user, you need to sign up for a HackerTarget membership. Once registered, you can find your API key in the user dashboard. This key can be included in your API requests either as a query parameter (&apikey=) or as a custom header (X-API-Key). (hackertarget.com)

By adhering to these rate limits and utilizing the appropriate authentication, you can effectively use HackerTarget's hostsearch API for your network security assessments.

Citations:

Allow Unauthenticated HackerTarget API Requests

HackerTarget’s hostsearch endpoint supports unauthenticated queries (up to 50 calls/day and 2 calls/sec per IP). Skipping entirely when no API key is present is overly restrictive—let’s fall back to an unauthenticated request instead.

Locations to update:

  • pkg/subscraping/sources/hackertarget/hackertarget.go: lines 34–42

Suggested diff:

-    randomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())
-    if randomApiKey == "" {
-        s.skipped = true
-        return
-    }
-
-    htSearchUrl = fmt.Sprintf("%s&apikey=%s", htSearchUrl, randomApiKey)
+    randomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())
+    if randomApiKey != "" {
+        // Authenticated request: unlimited results per plan
+        htSearchUrl = fmt.Sprintf("%s&apikey=%s", htSearchUrl, randomApiKey)
+    } else {
+        // Unauthenticated fallback: limited to 50 calls/day and 2 calls/sec
+        // Proceed without &apikey parameter
+    }

• Remove s.skipped = true and return to allow unauthenticated queries
• Optionally, add logging to warn when falling back to unauthenticated mode
• Be mindful of rate limits (50/day, 2/sec) to avoid HTTP 429 errors

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
htSearchUrl := fmt.Sprintf("https://api.hackertarget.com/hostsearch/?q=%s", domain)
randomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())
if randomApiKey == "" {
s.skipped = true
return
}
htSearchUrl = fmt.Sprintf("%s&apikey=%s", htSearchUrl, randomApiKey)
htSearchUrl := fmt.Sprintf("https://api.hackertarget.com/hostsearch/?q=%s", domain)
randomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())
if randomApiKey != "" {
// Authenticated request: unlimited results per plan
htSearchUrl = fmt.Sprintf("%s&apikey=%s", htSearchUrl, randomApiKey)
} else {
// Unauthenticated fallback: limited to 50 calls/day and 2 calls/sec
// Proceed without &apikey parameter
}
🤖 Prompt for AI Agents 
In pkg/subscraping/sources/hackertarget/hackertarget.go around lines 34 to 42,
the current code skips the request entirely if no API key is available, which is
too restrictive since HackerTarget allows unauthenticated queries with limits.
Remove the lines setting s.skipped to true and the return statement to allow the
request to proceed without an API key. Optionally, add a log warning indicating
fallback to unauthenticated mode. Ensure the URL is constructed without the
apikey parameter when no key is present, and consider implementing rate limiting
to respect the 50 calls/day and 2 calls/sec limits.

resp, err := session.SimpleGet(ctx, htSearchUrl)
if err != nil {
results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
s.errors++
Expand Down Expand Up @@ -78,14 +89,16 @@ func (s *Source) NeedsKey() bool {
return false
}

func (s *Source) AddApiKeys(_ []string) {
// no key needed
// TODO: env variable will not work if NeedsKey is false, entire api key management needs to be refactored
func (s *Source) AddApiKeys(keys []string) {
s.apiKeys = keys
}

func (s *Source) Statistics() subscraping.Statistics {
return subscraping.Statistics{
Errors: s.errors,
Results: s.results,
TimeTaken: s.timeTaken,
Skipped: s.skipped,
}
}
Loading