Prevent a netlas nil pointer dereference #1630

tekkamanendless · 2025-08-13T13:16:45Z

This uses two different variables for the two different HTTP requests, since two defer functions referenced the same resp variable.

This setup could cause a nil pointer dereference in the following scenario:

The first request succeeds (a defer on resp is added).
The second request fails (which sets resp to nil, and then the first defer attempts to reference resp.Body).

This change prevents that by not reusing the same variable for the second request.

closes #1634

Summary by CodeRabbit

Refactor
- Improved handling of sequential HTTP requests by separating response processing, ensuring clearer logic and more reliable resource cleanup.
- Updated status checks and response reading to align with distinct request phases, reducing risk of mixed states.
- No changes to user-facing behavior or functionality.

This uses two _different_ variables for the two _different_ HTTP requests, since two `defer` functions referenced the same `resp` variable. This setup could cause a nil pointer dereference in the following scenario: 1. The first request succeeds (a `defer` on `resp` is added). 2. The second request fails (which sets `resp` to nil, and then the first `defer` attempts to reference `resp.Body`). This change prevents that by not reusing the same variable for the second request.

coderabbitai · 2025-08-13T13:16:54Z

Walkthrough

Refactors HTTP handling in netlas.go by splitting a single response variable into two: resp1 for the initial GET (count) and resp2 for the POST (download). Updates status checks, body reads, and closures to reference the correct response object. No public interfaces or external behaviors changed.

Changes

Cohort / File(s)	Summary of Changes
Netlas dual-response refactor `pkg/subscraping/sources/netlas/netlas.go`	Replaced single resp with resp1 (GET count) and resp2 (POST download); updated status-code checks, body reads, and closures to the respective response; maintained 429 handling on resp2; no function signatures or exported entities modified.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~7 minutes

Poem

I nibble logs with tidy grace,
Two sips of HTTP I trace—
First a count, then files to chase.
Close the cups, no leaks to face,
Hop-hop! Clean streams in their place.
🐇✨

✨ Finishing Touches

📝 Generate Docstrings

🧪 Generate unit tests

Create PR with unit tests
Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

Visit our Status Page to check the current availability of CodeRabbit.
Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

coderabbitai

Actionable comments posted: 0

🧹 Nitpick comments (1)

pkg/subscraping/sources/netlas/netlas.go (1)
80-85: Consider using session.DiscardHTTPResponse for consistency.

Based on the retrieved learnings, the preferred pattern in subfinder sources is to use defer session.DiscardHTTPResponse(resp) after successful requests. While the current manual body closure approach works correctly, using the session's discard method would be more consistent with the codebase patterns.

Consider updating the defer statements:
 defer func() {
-	if err := resp1.Body.Close(); err != nil {
-		results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
-		s.errors++
-	}
+	session.DiscardHTTPResponse(resp1)
 }()
 defer func() {
-	if err := resp2.Body.Close(); err != nil {
-		results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
-		s.errors++
-	}
+	session.DiscardHTTPResponse(resp2)
 }()
Also applies to: 132-137

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 8feb51f and ae80dda.

📒 Files selected for processing (1)

pkg/subscraping/sources/netlas/netlas.go (4 hunks)

🧰 Additional context used

🧠 Learnings (3)

📓 Common learnings

Learnt from: x-stp
PR: projectdiscovery/subfinder#0
File: :0-0
Timestamp: 2025-06-24T16:03:30.467Z
Learning: When fixing HTTP response handling bugs in subfinder sources, the correct pattern is to use `defer session.DiscardHTTPResponse(resp)` after successful requests to ensure the response body remains open for reading, and call `session.DiscardHTTPResponse(resp)` immediately in error cases.

📚 Learning: 2025-07-17T12:07:51.521Z

Learnt from: 0x4500
PR: projectdiscovery/subfinder#1612
File: pkg/subscraping/agent.go:0-0
Timestamp: 2025-07-17T12:07:51.521Z
Learning: In subfinder subscraping sources, when a source needs to handle specific HTTP status codes differently (like treating 204 as success), the check should be implemented within the individual source's code rather than modifying the global httpRequestWrapper in agent.go. This keeps the special handling localized and avoids affecting other sources.

Applied to files:

pkg/subscraping/sources/netlas/netlas.go

📚 Learning: 2025-06-24T16:03:30.467Z

Learnt from: x-stp
PR: projectdiscovery/subfinder#0
File: :0-0
Timestamp: 2025-06-24T16:03:30.467Z
Learning: When fixing HTTP response handling bugs in subfinder sources, the correct pattern is to use `defer session.DiscardHTTPResponse(resp)` after successful requests to ensure the response body remains open for reading, and call `session.DiscardHTTPResponse(resp)` immediately in error cases.

Applied to files:

pkg/subscraping/sources/netlas/netlas.go

🧬 Code Graph Analysis (1)

pkg/subscraping/sources/netlas/netlas.go (11)

pkg/subscraping/sources/bevigil/bevigil.go (1)

Source (19-25)

pkg/subscraping/sources/bufferover/bufferover.go (1)

Source (25-31)

pkg/subscraping/sources/c99/c99.go (1)

Source (16-22)

pkg/subscraping/sources/censys/censys.go (1)

Source (57-63)

pkg/subscraping/sources/builtwith/builtwith.go (1)

Source (33-39)

pkg/subscraping/sources/chaos/chaos.go (1)

Source (14-20)

pkg/subscraping/sources/chinaz/chinaz.go (1)

Source (15-21)

pkg/subscraping/sources/certspotter/certspotter.go (1)

Source (20-26)

pkg/subscraping/sources/digitalyama/digitalyama.go (1)

Source (14-20)

pkg/subscraping/sources/dnsdumpster/dnsdumpster.go (1)

Source (23-29)

pkg/subscraping/sources/dnsdb/dnsdb.go (1)

Source (41-47)

🔇 Additional comments (8)

pkg/subscraping/sources/netlas/netlas.go (8)

66-69: LGTM! Correct fix for nil pointer dereference.

The separation of response variables (resp1 for the GET request, resp2 for the POST request) effectively prevents the nil pointer dereference scenario described in the PR objectives.

75-76: LGTM! Status code check correctly references the first response.

The status code check now properly references resp1.StatusCode instead of the previously shared resp variable.

81-81: LGTM! Body closure correctly references the first response.

The defer closure now properly references resp1.Body for the GET request response.

87-87: LGTM! Body reading correctly references the first response.

The body reading operation now properly references resp1.Body for the GET request response.

123-126: LGTM! Second HTTP request uses separate response variable.

The POST request now uses resp2 variable, preventing interference with the first request's response handling.

133-133: LGTM! Body closure correctly references the second response.

The defer closure now properly references resp2.Body for the POST request response.

138-138: LGTM! Body reading correctly references the second response.

The body reading operation now properly references resp2.Body for the POST request response.

145-146: LGTM! Status code checks correctly reference the second response.

The rate limiting check now properly references resp2.StatusCode for the POST request response.

zy9ard3 · 2025-08-15T12:39:06Z

+1

currently, I'm frequently experiencing this dereference issue while running subfinder ;

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0xfb92b9]

goroutine 3320 [running]:
github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/netlas.(*Source).Run.func1.2()
        /home/runner/work/subfinder/subfinder/v2/pkg/subscraping/sources/netlas/netlas.go:81 +0x19
github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/netlas.(*Source).Run.func1()
        /home/runner/work/subfinder/subfinder/v2/pkg/subscraping/sources/netlas/netlas.go:130 +0xe3b
created by github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/netlas.(*Source).Run in goroutine 3241
        /home/runner/work/subfinder/subfinder/v2/pkg/subscraping/sources/netlas/netlas.go:51 +0xe5

tekkamanendless · 2025-08-15T13:35:09Z

Note that the build tests are broken overall and can be fixed by #1631.

tekkamanendless · 2025-08-15T15:47:40Z

#1631 has been merged; everything except the macOS-13 test is passing now. No clue why the macOS one is failing.

dogancanbakir · 2025-08-26T08:28:44Z

Thanks for the PR!

coderabbitai bot reviewed Aug 13, 2025

View reviewed changes

Merge branch 'dev' into fix-netlas-panic

c7a3230

dogancanbakir merged commit b47737d into projectdiscovery:dev Aug 26, 2025
8 of 10 checks passed

BrewTestBot mentioned this pull request Sep 26, 2025

subfinder 2.9.0 Homebrew/homebrew-core#245805

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Prevent a netlas nil pointer dereference #1630

Prevent a netlas nil pointer dereference #1630

Uh oh!

tekkamanendless commented Aug 13, 2025 •

edited by dogancanbakir

Loading

Uh oh!

coderabbitai bot commented Aug 13, 2025 •

edited

Loading

Chat

Support

CodeRabbit Commands (Invoked using PR/Issue comments)

Other keywords and placeholders

CodeRabbit Configuration File (`.coderabbit.yaml`)

Status, Documentation and Community

Uh oh!

coderabbitai bot left a comment

Uh oh!

zy9ard3 commented Aug 15, 2025

Uh oh!

tekkamanendless commented Aug 15, 2025

Uh oh!

tekkamanendless commented Aug 15, 2025

Uh oh!

dogancanbakir commented Aug 26, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Prevent a netlas nil pointer dereference #1630

Prevent a netlas nil pointer dereference #1630

Uh oh!

Conversation

tekkamanendless commented Aug 13, 2025 • edited by dogancanbakir Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai bot commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Poem

Chat

Support

CodeRabbit Commands (Invoked using PR/Issue comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Status, Documentation and Community

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

zy9ard3 commented Aug 15, 2025

Uh oh!

tekkamanendless commented Aug 15, 2025

Uh oh!

tekkamanendless commented Aug 15, 2025

Uh oh!

dogancanbakir commented Aug 26, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tekkamanendless commented Aug 13, 2025 •

edited by dogancanbakir

Loading

coderabbitai bot commented Aug 13, 2025 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)