这是indexloc提供的服务,不要输入任何密码
Skip to content

Conversation

@dalf
Copy link
Contributor

@dalf dalf commented Nov 5, 2023

What does this PR do?

  • UWSGI_WORKERS specifies the number of process.
  • UWSGI_THREADS specifies the number of threads.

The Docker convention is to specify the whole configuration through environment variables. While not done in SearXNG, these two additional variables allows admins to skip uwsgi.ini

In additional, https://github.com/searxng/preview-environments starts Docker without additional files through searxng-helm-chat. Each instance consumes 1Go of RAM which is a lot especially when there are a lot of instances / pull requests. With UWSGI_WORKERS=1 each instance consume about 250Mo.

Why is this change important?

  • Easier docker configuration
  • Make preview-environments more reliable

How to test this PR locally?

  • make docker.build
  • docker run --rm -ti -p 8080:8080 ${GITHUB_LOGIN}/searxng should start as many workers as number of CPU cores (same as the current master branch).
  • docker run --rm -ti -p 8080:8080 -e UWSGI_WORKERS=1 ${GITHUB_LOGIN}/searxng should start one worker.

Author's checklist

Related issues

@dalf dalf changed the title Docker: add UWSGI_WORKERS and UWSGI_THREAD. Docker: add UWSGI_WORKERS and UWSGI_THREAD environment variables Nov 5, 2023
@dalf dalf requested a review from unixfox November 5, 2023 11:03
@unixfox
Copy link
Contributor

unixfox commented Nov 5, 2023

Can you add these new envs into the list: https://github.com/searxng/searxng/blob/master/Dockerfile#L18?

Thank you

@dalf dalf force-pushed the docker-add-env-var branch 2 times, most recently from 761cca9 to 9babefb Compare November 5, 2023 12:31
UWSGI_WORKERS specifies the number of process.
UWSGI_THREADS specifies the number of threads.

The Docker convention is to specify the whole configuration
through environment variables. While not done in SearXNG, these two
additional variables allows admins to skip uwsgi.ini

In additional, https://github.com/searxng/preview-environments starts Docker
without additional files through searxng-helm-chat.
Each instance consumes 1Go of RAM which is a lot especially when there are a
lot of instances / pull requests.
@dalf dalf force-pushed the docker-add-env-var branch from 9babefb to 17ca7eb Compare November 5, 2023 12:39
@dalf
Copy link
Contributor Author

dalf commented Nov 5, 2023

Done.

Important note: this work with new installations, older installation keep the previous version of uwsgi.ini. But at least it fixes the issue for the preview-environments.

- UWSGI_WORKERS specifies the number of process.
- UWSGI_THREADS specifies the number of threads.

Templates for uwsgi scripts can be tested by::

    UWSGI_WORKERS=8 UWSGI_THREADS=9 \
      ./utils/searxng.sh --cmd\
      eval "echo \"$(cat utils/templates/etc/uwsgi/*/searxng.ini*)\""\
      | grep "workers\|threads"

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
@return42
Copy link
Member

return42 commented Nov 5, 2023

Added a commit on top that adds the same environment to the script installation procedure.

One question I have: should we also set offload-threads = %k from UWSGI_THREAD environment?

AFAIK the offload-threads setting is only for static files:

@dalf
Copy link
Contributor Author

dalf commented Nov 5, 2023

The number of offload-threads never appeared to be an issue as far I know.
If we drop UWSGI in a near future, we won't have to explain this new env variable is not supported anymore.

@unixfox unixfox merged commit bd3f526 into searxng:master Nov 12, 2023
@return42
Copy link
Member

I think these lines can be also removed

patch_uwsgi_settings() {
CONF="$1"
# update uwsg.ini
sed -i \
-e "s|workers = .*|workers = ${UWSGI_WORKERS:-%k}|g" \
-e "s|threads = .*|threads = ${UWSGI_THREADS:-4}|g" \
"${CONF}"
}

I have seen these lines in my review, but I think we can't ...

The template is copied with out a replacement from the environment:

printf 'Create %s\n' "${CONF}"
cp "${REF_CONF}" "${CONF}"

Now we have a uwsg.ini with the raw strings $(UWSGI_WORKERS) & $(UWSGI_THREADS) in ..

workers = $(UWSGI_WORKERS)
# Number of threads per worker
# default value: 4 (see Dockerfile)
threads = $(UWSGI_THREADS)

After the copy this raw strings will be patched by patch_uwsgi_settings() function ..

printf 'Create %s\n' "${CONF}"
cp "${REF_CONF}" "${CONF}"
$PATCH_REF_CONF "${CONF}"

@dalf: after rethinking about theses lines; its no longer clear to me if it was intended.

When should $(UWSGI_WORKERS) & $(UWSGI_THREADS) be set .. from the environment when the uwsgi service is started? .. or from the patch_uwsgi_settings() when the service is installed?

.. can you have a look / thanks!

@unixfox
Copy link
Contributor

unixfox commented Nov 13, 2023

I think these lines can be also removed

patch_uwsgi_settings() {
CONF="$1"
# update uwsg.ini
sed -i \
-e "s|workers = .*|workers = ${UWSGI_WORKERS:-%k}|g" \
-e "s|threads = .*|threads = ${UWSGI_THREADS:-4}|g" \
"${CONF}"
}

(reposting with the correct account)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants