Using a Rust/C(pp)/Go/Zig module for batched networking and encryption #5003
Replies: 2 comments 2 replies
-
|
Its not such easy as you might think .. in SearXNG the outgoing request are using this implementation of a network stack which is based on httpx (async). This "network stack" serves a lot of features, including proxies and more / see Lines 1193 to 1214 in d8c74c8 On the long term we consider to implement an addition network stack, may based on curl .. but this is nothing we have planed in the near future. |
Beta Was this translation helpful? Give feedback.
-
|
I understand. But has this been profiled? Do we know how much time is spent parsing http and passing around variables between the ssl implementation and the http parser versus everything else that is done for each request? |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Most of the networking SearXNG does is bog standard HTTP requests and can be batched. I.e send requests to this list of search engines and return the response body to parse. It reduces the time spent in Python's runtime under GC and can benefit from aggressively optimized HTTP/TLS/TCP/QUIC(?) implementations with no type conversion requirements. It also makes it easier to tune TLS/TCP parameters to handle fingerprinting.
Beta Was this translation helpful? Give feedback.
All reactions