这是indexloc提供的服务,不要输入任何密码
Skip to content

Conversation

@HashimTheArab
Copy link
Contributor

@HashimTheArab HashimTheArab commented Sep 5, 2025

  • Proper overflow detection for limit instead of just returning truncated data
  • Fixed decompression pool usage, old logic put reader back in pool without closing in cases of Reset errors
  • Old logic closed the reader before reading which worked but was wrong
  • Improved capacity hints, does not set to * 2 if its not needed
  • Significant memory usage improvements to reduce allocations by using bufferpool

Comment on lines 124 to 128
capHint := limit
if l <= limit/2 {
capHint = l * 2
}
decompressed.Grow(capHint)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this can be simplified to:

decompressed := bytes.NewBuffer(make([]byte, 0, min(limit, len(compressed) * 2)))

Copy link
Contributor Author

@HashimTheArab HashimTheArab Sep 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The if checking the l <= limit/2 protects against an int overflow on 32 bit devices if a packet is 1gb, which is impossible but having the if instead of min is technically correct

Comment on lines 130 to 136
// Handle no limit
if limit == math.MaxInt {
if _, err := io.Copy(&decompressed, r); err != nil {
return nil, fmt.Errorf("decompress flate: %w", err)
}
return decompressed.Bytes(), nil
}
Copy link
Owner

@Sandertv Sandertv Sep 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there ever a sensible case to have no limit? I get that this is probably to avoid overflow, but could just do a max(limit + 1, math.MaxInt) somewhere.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

limit + 1 would cause an overflow so we cant handle this without checking limit == math.MaxInt, which is what the listener sets if the user configures no limit

cfg.MaxDecompressedLen = math.MaxInt

@HashimTheArab HashimTheArab changed the title fix: flate decompression logic Improve flate decompression logic and significantly improve memory usage Oct 28, 2025
Copy link
Owner

@Sandertv Sandertv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these changes look sensible. I assume this has been tested thoroughly?

@Sandertv Sandertv merged commit 8ae798f into Sandertv:master Oct 29, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants