TLDR: are there use cases for the multi_requests function other than the baidu engine?

A note about async HTTP client and the multi_requests function

As its name says, the function sends multiple requests at the same time. The function returns all the responses at once.
The implementation relies on an async HTTP client to send all the HTTP requests in parallel. This is the perfect use case for async.

All other HTTP requests can run in the sync world: there is no parallelism in the engine code.

However, in the master branch, all outgoing HTTP requests are sent through async HTTP clients. There are bridges from sync to async and vice versa. There was at a time when SearX(NG) was supposed to switch async entirely: https://github.com/searx/searx/wiki/Milestones#milestone-12---async
This has never happened and will not.

Async is (nearly|for sure) an anti-pattern in a Flask application: if we need parallelism, the way to go is thread.

Also, async makes the HTTP streaming complex. After each change, I ask someone on the team (usually Paul) to deploy it on a public instance and monitor for memory leaks.

So #2685 drops all async code and relies only sync HTTP client ... and the multi_requests is dropped too: this function is not used (the Bing engine used it, but not anymore).

The only potential use case for the multi_requests function is the Baidu engine. The URL returns by Baidu are something like http://www.baidu.com/link?url=. We can implement multi_requests using only threads: each call creates one thread per URL. In this scenario, one user query needs about 11 HTTP requests (1 for the search, and about 10 for the results) = 11 threads per user request. If there are ten user queries simultaneously, that's about 110 threads.
I don't know if this is an problem to create a lot of thread. If it is:

• either we says that we can't implement the Baidu engine and similar engine (currently, no other engine).
• or we find a way to implement the multi_requests.

So my question is: Are there other use cases for the multi_requests function?

I'm just wondering what failed for the previous attempts and why they failed.

Disclaimer:

• I’ve tried to use async multiple times, but my tests lack thorough review. It was a few years ago, and the situation may have changed with improvements in newer Python versions.
• I aim to summarize my tests here, but I will likely forget some details and might add an edit section to this message (or just add another comment).
• Comparing the performance between two code versions is challenging. The tests require the same configuration, the same bare-metal hardware, and various workload types with automated tests. I haven’t done this as meticulously as needed to make a definitive decision in most of the cases.

My main concern about full async (web framework, HTTP client, engines) : if one engine is slow, everything else slows down. I ended up with some weird things like this to improve the response time. With a single thread to handle async code (by design in Python), we have to be very careful everywhere (also related to my comment on httpx below).

Few years ago I though about this architecture:

• async for the web framework and HTTP clients
• sync code for the engines running in a pool of workers
• async and sync codes communicate using queues.

... this does not fit with the concept of HTTP server for Python app that take control of the app (like uwsgi and all others).

Web framework

Quart prototype

searx/searx#1724

Comments to take with pinch of salt, from what I remember:

• async jinja2 rendering was slower. Note: two years after this POC, there is a fixed related issue: Async is slow due to isawaitable use pallets/jinja#1514
• Quart response times were slower than the Flask version. I don't remember clearly if I have test a huge workload.

I know this is not quantitative...

Starlette:

• [WIP] Starlette searx/searx#2313
• [experimental] Starlette #256

There is no proper support for babel: Kludex/starlette#279

Client

httpx

httpx is a good example of well written code and carefully tested, however the async code has a lot of await for each chunk of data especially with HTTP/2. According to my tests, it is fast enough for few kilobytes like a search engine answer. I remember a test on @mrpaulblack 's instance with HTTP/2 for the image proxy : the CPU usage increased by a lot --> things went back to normal when HTTP/2 was disabled for the image proxy (this is the default settings now).

4 years ago, I tried to benchmark httpx : https://github.com/dalf/pyhttp-benchmark/blob/master/results/output.md
The test is old, it would be good to test again.

Without success, I've tried various things to improve the speed:

• compile httpx with Cython
• partially rely on nghttp2 : https://github.com/dalf/hpack_nghttp2/
• Most probably some other attempts I forgot

I've seen that homeassistant use zlib-ng, perhaps this can help.

pycurl

searx/searx#1725

pycurl is a binding to the system libcurl, and there are a lot of different version of libcurl. I've stopped the POC when I encountered a segfault using HTTP/2 on Ubuntu 18.04.

That was a six years ago, perhaps things has changed since then.

aiohttp

#254

There is no support for HTTP/2 which really help to avoid CAPTCHA for some engines (I don't remember which one, @unixfox has a better knowledge on this topic).

Actually we could use this for the image proxy.

HTTP server

Related, more or less out of topic: perhaps, a good replacement for uwsgi is granian which support WSGI or ASGI apps.