这是indexloc提供的服务,不要输入任何密码
Skip to content

Conversation

@inetol
Copy link
Member

@inetol inetol commented Mar 6, 2025

Related #4378

Closes #965

To be merged after #4424

@vojkovic
Copy link
Member

vojkovic commented Mar 7, 2025

How does it listen to ipv4 as well? You've only made it bind to the v6 wildcard address. This will probably confuse people when debugging they can't reach anything over v4.

@inetol
Copy link
Member Author

inetol commented Mar 7, 2025

How does it listen to ipv4 as well? You've only made it bind to the v6 wildcard address. This will probably confuse people when debugging they can't reach anything over v4.

Applications may use AF_INET6 sockets to open TCP connections to IPv4
nodes, or send UDP packets to IPv4 nodes, by simply encoding the
destination's IPv4 address as an IPv4-mapped IPv6 address, and
passing that address, within a sockaddr_in6 structure, in the
connect() or sendto() call.

https://www.rfc-editor.org/rfc/rfc3493#section-3.7

..basically, every operating system or thing that supports IPv6 will automatically convert from IPv4 to IPv6 space for you.

And if someone explicitly defines the BIND_ADDRESS and the dual stack screwed it is not our problem, all applications go and will go through this transition process someday and users need to learn how to deal with it

@unixfox
Copy link
Contributor

unixfox commented Mar 7, 2025

In most programming language like python, [::] means that the app will open a socket on both IPv4 and IPv6.

@vojkovic
Copy link
Member

vojkovic commented Mar 7, 2025

Ah interesting - in golang it would just listen to v6. Learned something new about python today :)

Ok, so after reading the RFC, IPv4 clients should appear as ::ffff:192.0.2.1 in the logs for example?

I'll take a in depth look later tonight.

@unixfox
Copy link
Contributor

unixfox commented Mar 7, 2025

Ok well maybe not in python about the sockets listening on both ipv4/ipv6. But indeed it's going to do what the RFC say.

If do I:

python -m http.server -b :: 8080
curl 127.0.0.1:8080

Then I get:

::ffff:127.0.0.1 - - [07/Mar/2025 12:04:48] "GET / HTTP/1.1" 200 -

Now I that I think about it, I wonder if this could mess up with the rate limiter that doesn't expect IPv4 IP addresses into an IPv6 address.

According to stackoverflow, if we want to have a dual stack socket then we need to use dualstack_ipv6=True: https://stackoverflow.com/questions/65877630/how-to-bind-both-ipv4-and-ipv6-source-address-to-python-socket

In Docker I thought it would "remove" the ::ffff but seems like it is not:

docker run -p 8080:8080 python:3-slim python3 -m http.server -b :: 8080
curl 127.0.0.1:8080
::ffff:172.17.0.1 - - [07/Mar/2025 11:11:43] "GET / HTTP/1.1" 200 -

@inetol
Copy link
Member Author

inetol commented Mar 7, 2025

In Docker I thought it would "remove" the ::ffff but seems like it is not:

In fact, this is expected to identify that an IPv6 address comes from IPv4 (RFC3493; section 3.7), although now that you mention about ratelimit this may be an issue

@unixfox
Copy link
Contributor

unixfox commented Mar 7, 2025

@return42 would you be ok to add a specific case about ::ffff IPv6 address in the limiter?

I guess we would only need to catch this specific :ffff: and then remove the :ffff: part in order to get the wanted IPv4 address.

@return42
Copy link
Member

return42 commented Mar 7, 2025

@return42 would you be ok to add a specific case about ::ffff IPv6 address in the limiter?

By default IPv4Address.is_link_local are not filtered:

if network.is_link_local and not cfg['botdetection.ip_limit.filter_link_local']:
logger.debug("network %s is link-local -> not monitored by ip_limit method", network.compressed)
return None

Or with other words, by default botdetection.ip_limit.filter_link_local is set to false.

@unixfox
Copy link
Contributor

unixfox commented Mar 7, 2025

@return42 if we merge this PR, all the IPv4 address space will be like this for Docker.

So my question is not only about the localhost IP addresses but all the IPv4 address space.

@return42
Copy link
Member

return42 commented Mar 7, 2025

So my question is not only about the localhost IP addresses but all the IPv4 address space.

Python has an abstraction named ip_address.

A ::ffff: prefixed IP is handled as (any other) IPv6 address:

>>> from ipaddress import ip_address
>>> ip_address("::ffff:192.0.2.1")
IPv6Address('::ffff:192.0.2.1')

while this is a IPv4 address ..

>>> ip_address("192.0.2.1")
IPv4Address('192.0.2.1')

@vojkovic
Copy link
Member

vojkovic commented Mar 7, 2025

By default IPv4Address.is_link_local are not filtered:

Keep in mind that IPv4-mapped addresses aren't link local, they are from the block ::ffff:0:0/96 whilst link local are from fe80::/10 - they won't be exempt from the limiter by default.

Either way the limiter must be able to differentiate between v4 and v6 so that IP ranges can be properly blocked. Because this range is treated like any other v6 address, blocking a /48 (default) would ban all of IPv4 address space.


ipv6_prefix = 48

@return42
Copy link
Member

return42 commented Mar 7, 2025

The limiter uses this function to get the IP address ..

def get_real_ip(request: SXNG_Request) -> str:
"""Returns real IP of the request. Since not all proxies set all the HTTP
headers and incoming headers can be faked it may happen that the IP cannot
be determined correctly.

Either way the limiter must be able to differentiate between v4 and v6 so that IP ranges can be properly blocked. Because this range is treated like any other v6 address, blocking a /48 (default) would ban all of IPv4 address space.

May we can (as @unixfox suggested) remove a :ffff: prefix before returning the request_ip?

request_ip = forwarded_for or real_ip or remote_addr or '0.0.0.0'
# logger.debug("get_real_ip() -> %s", request_ip)
return request_ip

@vojkovic
Copy link
Member

vojkovic commented Mar 7, 2025

May we can (as @unixfox suggested) remove a :ffff: prefix before returning the request_ip?

Yes, this will be the best place to do it (I recommend using ipv6_mapped though).

We should also consider where else we're using remote_addr - such as in the Tor Check plugin - which this PR would break for v4.

Actually after looking at this, I think the tor check plugin should use get_real_ip() instead - would you be happy if I send through a PR for this done?

if x_forwarded_for:
ip_address = x_forwarded_for[0]
else:
ip_address = request.remote_addr

@return42
Copy link
Member

return42 commented Mar 7, 2025

would you be happy if I send through a PR for this done?

FYI I incorporated the patch in PR:

@unixfox
Copy link
Contributor

unixfox commented Mar 7, 2025

I vouch on changing the bind address for the settings.yml too:

bind_address: "0.0.0.0"

Is everyone ok?

@unixfox
Copy link
Contributor

unixfox commented Mar 8, 2025

@inetol could you add that too as a modification in this PR :)?

you can change the title like "add default support for IPv6"

@inetol inetol changed the title container: uWSGI listen both ip6 & ip4 container: add default support for IPv6 Mar 8, 2025
@unixfox
Copy link
Contributor

unixfox commented Mar 8, 2025

would you be happy if I send through a PR for this done?

FYI I incorporated the patch in PR:

* [[refactor] typification of SearXNG (MainResult) / result items (part 2) #4424](https://github.com/searxng/searxng/pull/4424)

@return42 May I understand that what I have asked you is resolved in the PR above?

@return42 would you be ok to add a specific case about ::ffff IPv6 address in the limiter?

I guess we would only need to catch this specific :ffff: and then remove the :ffff: part in order to get the wanted IPv4 address.

Thank you in advance :)

@return42
Copy link
Member

return42 commented Mar 9, 2025

@return42 May I understand that what I have asked you is resolved in the PR above?

yes / sorry if it was not clear :-)

@unixfox
Copy link
Contributor

unixfox commented Mar 9, 2025

Great, I guess we can wait for #4424 to be merged in order to merge this PR (#4448).

@return42
Copy link
Member

@return42 May I understand that what I have asked you is resolved in the PR above?

yes / sorry if it was not clear :-)

FYI: PR is now merged .. #4424

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Docker image: the default_bind_address in docker-entrypoint.sh takes over and forces ipv4.

4 participants