[mod] rework container CI/CD #4707

inetol · 2025-05-02T11:50:09Z

What does this PR do?

This is one of various PR to refactor the entire SearXNG Docker workflow.

Refactors integration.yml and moves container related things into container.yml, also touches needed updates into script files.

Podman is now supported to build the container images (Docker also received a refactor, merging both build and buildx)
Bringing massive reduction on runtime across all CI/CD workflows.
Container images are being built by Buildah instead of Docker BuildKit.
Container images are being tested before release.
Splitting "modern" (amd64 & arm64) and "legacy" (armv7) arches on different Dockerfiles to reduce dependency burden and allowing future image optimizations.

Why is this change important?

Same as #4699

How to test this PR locally?

make container.build

Author's checklist

We can't verify locally that the CI process works as expected, especially the container image release process. Due to the changes being relatively large, it's likely that some workflow will not work correctly the first time.

I try as much as possible to verify that everything works individually, but no promises, CI stuff.

return42 · 2025-05-03T09:16:41Z

Due to my limited access to the org .. I try as much as possible to verify that everything works individually, but no promises, CI stuff.

Thank you very much for the work you put in here ... if you get stuck at some point, don't hesitate to ask the core team for support. I myself am weak on the topics of GH actions and docker ... but there are others in our team and as far as I know they are very familiar with them ... and can also decide whether we should grant you additional rights on the GH repository.

unixfox · 2025-05-04T20:48:14Z

Just would like to point out that every commit creates a notification on our side. It would maybe be interesting to test in a fork on your side. Thanks

inetol · 2025-05-04T20:53:36Z

Just would like to point out that every commit creates a notification on our side. It would maybe be interesting to test in a fork on your side. Thanks

Sorry about that, I will try to move the tests directly to the fork, but I don't know if the CI will work as expected.

inetol · 2025-05-07T21:35:11Z

There should be no more major changes to this PR unless requested. I have updated the PR information with the most significant changes #4707 (comment)

Take your time to read the diff and request changes.

Copilot

Pull Request Overview

This PR refactors the CI/CD workflows and reworks the container build process to support Podman and Buildah while improving runtime efficiency. Key changes include a redesigned container script (lib_sxng_container.sh) to merge Docker and Podman builds, updates to workflow YAML files for enhanced caching and concurrent execution, and adjustments in version handling within searx/version.py.

Reviewed Changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

File	Description
utils/lib_sxng_container.sh	Refactored container build, test, and push functions; improved engine selection and architecture handling
searx/version.py	Updated version resolution to handle GitHub Actions environment variables
manage, Makefile, Dockerfiles, workflows	Updated command aliases, caching keys, and YAML definitions to streamline CI/CD processes

utils/lib_sxng_container.sh

return42 · 2025-05-08T04:37:30Z

manage

    case "$cmd" in
-        --getenv) var="$1"; echo "${!var}";;
-        --help) help;;
-        --*)
-            help
-            err_msg "unknown option $cmd"
+    --getenv)
+        var="$1"
+        echo "${!var}"
+        ;;
+    --help) help ;;
+    --*)
+        help
+        err_msg "unknown option $cmd"
+        return 42
+        ;;
+    *)
+        _type="$(type -t "$cmd")"
+        if [ "$_type" != 'function' ]; then


I think these modifications here are only whitespaces .. may related to your editor settings. Some editors interpret one tab as 4 spaces (a tab is by definition 8 spaces). To avoid tab problem, we don't use tab for indentation (except in Makefiles, where a tab is needed).

For shell files we use 4 spaces for one indentation:

searxng/.editorconfig

Lines 5 to 7 in bc06b1a

[*]

indent_style = space

indent_size = 4

Can you please check your editor settings.

The editor forces me to use the style defined in .editorconfig and indeed those are spaces and not tabs.

What I do use is https://github.com/mvdan/sh and it seems that by default it flattens the case branches, that's why the diff

utils/lib_sxng_container.sh

return42 · 2025-05-08T04:53:54Z

manage

        PLATFORM="$(python -c 'import platform; print(platform.system().lower(), platform.architecture()[0])')"
        case "$PLATFORM" in
-            "linux 32bit" | "linux2 32bit") ARCH="linux32";;
-            "linux 64bit" | "linux2 64bit") ARCH="linux64";;
-            "windows 32 bit") ARCH="win32";;
-            "windows 64 bit") ARCH="win64";;
-            "mac 64bit") ARCH="macos";;
+        "linux 32bit" | "linux2 32bit") ARCH="linux32" ;;
+        "linux 64bit" | "linux2 64bit") ARCH="linux64" ;;
+        "windows 32 bit") ARCH="win32" ;;
+        "windows 64 bit") ARCH="win64" ;;
+        "mac 64bit") ARCH="macos" ;;
        esac
-        GECKODRIVER_URL="https://github.com/mozilla/geckodriver/releases/download/$GECKODRIVER_VERSION/geckodriver-$GECKODRIVER_VERSION-$ARCH.tar.gz";
+        GECKODRIVER_URL="https://github.com/mozilla/geckodriver/releases/download/$GECKODRIVER_VERSION/geckodriver-$GECKODRIVER_VERSION-$ARCH.tar.gz"



🤔 do you use a code optimizer tool / code formating tool ? --> except the removed docker function all other modifications in this script are not needed.

If you know a code formating tool for shell scripts, and you think its worth to add to our tool chain, then please open a new PR for such a task.

Should I revert those style change for now? I'll open a PR later with the tool.

Should I revert those style change for now?

Yes, it is better to not have this modification in this PR / they are not needed to achieve the goal of this PR.

I'll open a PR later with the tool.

👍

return42 · 2025-05-08T05:23:45Z

The PR brings many valuable changes, but it also combines too many changes. It will therefore be difficult for any reviewer to approve this PR in its entirety.

I would suggest creating smaller PRs that cherry pick topics from this DRAFT. I would therefore suggest to create

one PR for the build of the online doc (documentation.yml)
one PR for translation tasks (l10n.yml)
one PR for docker build tasks (the name container.yml does not fit, since it is not about containers in general but docker in particular)
...

return42 · 2025-05-08T06:14:27Z

TL;DR / the follwing rules should be in mind, when creating a commit:

commit history should be read like a history book
commit messages are for the reader not for the author of the commit

There should be no more major changes to this PR unless requested.

This PR has 57 commits .. but not a single one is related to a separate topic ... some commit messages have titles like . or .. ... strictly speaking they are not commits but saves of files (like: "save my files to the repo").

A commit should always have a context and the commit message describes what is to be changed in that context, just as a function description should describe what the intention and the goal of the function is, a commit message should describe what the intention and the goal of that commit is.

Choose meaningful commit messages](https://docs.searxng.org/dev/contribution_guide.html#code)

The commits are not used to save files.

The commit messages form the history and are the first and therefore most important information a developer has when he has to research when and why a change had to be made and how it was made (what the goal was).

Like any text, a commit message should be written for the reader and not from the perspective of the author.

When scrolling through the history, the first thing one see is the title of the commit message. Therefore the title should describe the change as briefly and precisely as possible ... followed by a blank line and then a somewhat detailed description of the change.

Since the 57 commits are of no further significance, they could be combined into one commit ... or you could take the trouble to divide them into topics, but since we probably have to make new (individual) PRs for the topics anyway, we can improve the commit history even then.

inetol · 2025-05-08T09:25:16Z

This PR has 57 commits .. but not a single one is related to a separate topic ... some commit messages have titles like . or .. ... strictly speaking they are not commits but saves of files (like: "save my files to the repo").

Yeah, most of them come from another branch of this fork. I'll squash and document them now.

Also fixes a cache issue found on data-update.yml, this fix is applied on checker.yml and container.yml files

Again. Needed to test the workflows in a separate fork so unrelated triggers appear in files. Also, adds the new Dockerfile.compat for armv7.

Needed to test the workflows in a separate fork so unrelated triggers appear in files. Run tests before container release, if there are any problems the job fails and the images are not released. At the moment this is basic, but it should cover the most serious issues that may appear. This will prevent fiascos like searxng#4718 to ever happen again (hopefully)

concurrency rules applies to each branch

When running the version.py in CI, it only fetches a detached HEAD failing to get the reference. As I do not know very well if modifying the function will affect "normal" cases, I have preferred to use the variables provided by GHA to obtain the branch and the location of the repository.

preparation for initial review, moves all container stuff to lib_sxng_container.sh and reverts changes from fork CI testing

searxng#4707 (comment) searxng#4707 (comment) From Copilot review, manually tested

Dockerfile

dalf · 2025-05-09T06:08:45Z

.github/workflows/checker.yml

-  checker:
-    name: Checker
-    runs-on: ubuntu-24.04
+  search:


Note:

the Python module name should be searx.checker rather than searx.search.checker.

for a time it made sense to embed this tool in the Flask app, I think now it could move to external tool.

dalf · 2025-05-09T06:25:35Z

Dockerfile.compat

@@ -0,0 +1,106 @@
+FROM docker.io/library/python:3.13-slim AS builder


This might be common knowledge, but adding a comment at the top of the file to indicate that this Dockerfile is intended for ARMv7 would be helpful for new comers.

If we want to avoid to duplicate information (ARMv7 here in comment, ARMv7 in code elsewhere), we can add a comment stating that utils/lib_sxng_container.sh pick this file for some architectures.

I've renamed the Dockerfile.compat to armv7.Dockerfile here https://github.com/searxng/searxng/pull/4764/files#diff-92d87db7221875202a920aefc797462a7edac78c8ee2c172540cb2876811be75

inetol · 2025-05-10T11:53:42Z

I'll keep the commit devel history here.

This PR is superseded by #4764

This comment was marked as resolved.

Sign in to view

inetol mentioned this pull request May 4, 2025

Rework Dockerfile #4699

Merged

inetol changed the title ~~Rework Docker release workflow~~ Rework CI/CD workflows May 4, 2025

inetol force-pushed the container-gha branch from c2342f7 to 28241e3 Compare May 4, 2025 20:48

This was referenced May 4, 2025

Rework CI/CD workflows (duo) inetol/searxng#1

Merged

Push a container to ghcr.io because of new DockerHub rate limits #4374

Closed

OCI image format on containers #4722

Closed

inetol force-pushed the container-gha branch 2 times, most recently from e01bb93 to 6f5cf72 Compare May 7, 2025 20:58

inetol requested review from Copilot, dalf and return42 May 7, 2025 21:35

Copilot AI reviewed May 7, 2025

View reviewed changes

utils/lib_sxng_container.sh Outdated Show resolved Hide resolved

utils/lib_sxng_container.sh Outdated Show resolved Hide resolved

inetol changed the title ~~Rework CI/CD workflows~~ [enh] Rework CI/CD workflows May 7, 2025

return42 reviewed May 8, 2025

View reviewed changes

utils/lib_sxng_container.sh Show resolved Hide resolved

return42 reviewed May 8, 2025

View reviewed changes

inetol added 6 commits May 8, 2025 11:35

Initial update

9b8517a

refactor checker.yml

bbbd61e

refactor data-update.yml

a7efdd9

Also fixes a cache issue found on data-update.yml, this fix is applied on checker.yml and container.yml files

refactor integration.yml

8f5296e

refactor security.yml

0969926

compat Dockerfile & misc fixes

ee28f8e

Again. Needed to test the workflows in a separate fork so unrelated triggers appear in files. Also, adds the new Dockerfile.compat for armv7.

inetol added 6 commits May 8, 2025 11:44

run workflows sequentially by default

e37bd44

concurrency rules applies to each branch

migrate image release process

be04143

cleanup

4314b0e

preparation for initial review, moves all container stuff to lib_sxng_container.sh and reverts changes from fork CI testing

copilot review changes

af5a3db

searxng#4707 (comment) searxng#4707 (comment) From Copilot review, manually tested

inetol force-pushed the container-gha branch from 87d5000 to af5a3db Compare May 8, 2025 10:01

inetol mentioned this pull request May 8, 2025

[enh] refactor checker.yml workflow #4730

Merged

inetol removed the request for review from dalf May 8, 2025 10:22

This was referenced May 8, 2025

[enh] refactor security.yml workflow #4731

Merged

[enh] refactor data-update.yml workflow #4732

Merged

[mod] new documentation.yml workflow #4733

Merged

[mod] new l10n.yml workflow #4734

Merged

inetol changed the title ~~[enh] Rework CI/CD workflows~~ [mod] rework container CI/CD May 8, 2025

unixfox suggested changes May 9, 2025

View reviewed changes

Dockerfile Show resolved Hide resolved

dalf reviewed May 10, 2025

View reviewed changes

This was referenced May 10, 2025

[mod] refactor integration.yml workflow #4763

Merged

[mod] rework container deployment #4764

Merged

inetol closed this May 10, 2025

return42 mentioned this pull request May 11, 2025

Checker cleanup / restructure #4765

Open

inetol deleted the container-gha branch May 12, 2025 08:13

return42 mentioned this pull request May 18, 2025

Add shell (bash) formatter to SearXNG's tool chain #4803

Closed

		@@ -0,0 +1,106 @@
		FROM docker.io/library/python:3.13-slim AS builder

[mod] rework container CI/CD #4707

[mod] rework container CI/CD #4707

Uh oh!

Conversation

inetol commented May 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is this change important?

How to test this PR locally?

Author's checklist

Uh oh!

This comment was marked as resolved.

return42 commented May 3, 2025

Uh oh!

unixfox commented May 4, 2025

Uh oh!

inetol commented May 4, 2025

Uh oh!

inetol commented May 7, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

return42 commented May 8, 2025

Uh oh!

return42 commented May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

inetol commented May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

inetol commented May 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

inetol commented May 2, 2025 •

edited

Loading

return42 commented May 8, 2025 •

edited

Loading

inetol commented May 8, 2025 •

edited

Loading