[enh] py: drop deps #5407
[enh] py: drop deps #5407
Conversation
| pyyaml==6.0.3 | ||
| httpx[http2]==0.28.1 | ||
| httpx-socks[asyncio]==0.10.0 | ||
| Brotli==1.1.0 |
There was a problem hiding this comment.
As far I remember this Brotli requirement was added in first place to the httpx stack, to process incoming HTTP responses:
However, I don't know if there is any engine that absolutely requires Brotli. Usually, (http incomming) compression can be controlled via the Accept-Encoding header.
The default is:
Line 1043 in 0245327
If we remove Brotli here (what is OK for me), we should also have a look at the Accept-Encoding header in our outgoing requests and remove the br
searxng/searx/engines/tineye.py
Line 76 in 0245327
Other places can be found by the following search query:
There was a problem hiding this comment.
There's not a single site that doesn't support uncompressed responses, so sites not serving br content they will fall back on something else or uncompressed, but I don't understand why Accept-Encoding header is overridden on some engines. httpx will set the supported encodings automatically, as every HTTP client does.
This looks like a valid exception:
searxng/searx/engines/brave.py
Lines 201 to 202 in 0245327
...The other matches are not.
There was a problem hiding this comment.
why Accept-Encoding header is overridden on some engines.
..
httpx will set the supported encodings automatically, as every HTTP client does.
I guess it was my fail .. when reverse engineering these engines, I simply copied the header from my browser without considering that SearXNG (or httpx) already sets a default value for it.
the supported encodings automatically, as every HTTP client does.
Can you please also correct the unnecessary and partially incorrect overwriting of the headers in this branch? Thanks 👍
update: you already cleaned up the header while I had written the lines above :-)
The difference between decompression with brotli or gzip in HTML files is negligible for 3 MB of compiled binary package. Introduced in searxng@eaa694f Closes https://github.com/searxng/searxng/security/code-scanning/276 Closes https://github.com/searxng/searxng/security/dependabot/37
return42
left a comment
There was a problem hiding this comment.
Sorry, totally forgot this PR .. thanks for cleaning the HTTP header.
The difference between decompression with brotli or gzip in HTML files is negligible for 3 MB of compiled binary package.
unixthreadnamelooks like a weird workaround to me, this wasn't handled on uWSGI/Granian side?Introduced in eaa694f
Closes https://github.com/searxng/searxng/security/code-scanning/276
Closes https://github.com/searxng/searxng/security/dependabot/37