[fix] botdetection: check if the browser supports Sec-Fetch headers #4696

unixfox · 2025-04-30T09:14:11Z

Related to #3965.

Check that the browsers checked do support the feature Sec-Fetch before blocking them based on this factor.

Avoiding blocking legitimate users using ancient browsers that do not support these headers.

Bnyro

I think that some bots use Firefox user agents, so this should be helpful for such.

I'm not sure if the general impact is as good as with @return42's PR because many bots will probably use other user agents than the common ones (or rotate different ones) - we'll have to see how it performs in production.

unixfox · 2025-04-30T17:21:32Z

@return42 could you try on your searxng instance please :)?

unixfox · 2025-05-02T13:34:49Z

@return42 did you see any change in efficiency between PR #3965 and #4696 on your server?

return42 · 2025-05-02T13:39:17Z

@unixfox please give me some more time ..

I'm not sure if the general impact is as good as with @return42's PR because many bots will probably use other user agents than the common ones (or rotate different ones) - we'll have to see how it performs in production.

I want to implement an option for the SearXNG admins to switch additional browser tests from 399a31f on or off ...

unixfox · 2025-05-02T13:46:59Z

What do you mean? I'm not following you.

All the browsers in https://github.com/searxng/searxng/pull/4696/files#diff-ab6d5589d48207e72a940dbaa37d671093e2e1e7bc9849faed4e3163ed72a1e7R41-R76 match all the browsers that exist and support this Sec-Fetch headers. Firefox, Chrome, Safari.

return42 · 2025-05-02T14:02:00Z

Sorry, I quoted the wrong paragraph 🤦 .. I fixed my comment above to:

I'm not sure if the general impact is as good as with @return42's PR because many bots will probably use other user agents than the common ones (or rotate different ones) - we'll have to see how it performs in production.

unixfox · 2025-05-02T14:48:45Z

I want to implement an option for the SearXNG admins to switch additional browser tests from 399a31f on or off ...

Again I'm not following what you want to do. This is not clear whenever you want to implement an option to enable the feature "HTTP Fetch Metadata Request Headers" from PR #4696 or #3965. Or give the ability to add more browsers to check against "HTTP Fetch Metadata Request Headers".

If it's the latter one: why? Only the browsers checked in https://github.com/searxng/searxng/pull/4696/files#diff-ab6d5589d48207e72a940dbaa37d671093e2e1e7bc9849faed4e3163ed72a1e7R41-R76, do support Sec-Fetch. Why giving the ability to configure more? If there are more later on, then we can add them in the code.

return42 · 2025-05-02T14:58:37Z

many bots will probably use other user agents

The PR #3965 doesn't care UA header and blocks when Sec header is missed, while this PR looks at UA header and blocks only when UA header and Sec header do not fit together .. I want to implement an option, so admin can decide if he wants to care UA header or not.

unixfox · 2025-05-02T15:08:15Z

I want to implement an option, so admin can decide if he wants to care UA header or not.

I'm sorry I do not want this option. Here is why:

I do not want all the public instances of SearXNG to only work on these browsers:

Chrome >= 80
Firefox >= 90
Safari >= 16.4

Feel free to do that in your fork, on your instance, but I do not want such thing in the general SearXNG core.

Even if this is going to not be a default option, public instances owners will enable it because they want the best protection, and they will not understand the damages this is going to do.

My PR as it is a good balance to block based on one more factor (that remember, can always be found by reading the source code) and avoid blocking legitimate users.

So please, test it on your instance, give me some feedback for the efficiency in blocking bots (not legitimate users, if that can be analyzed).

return42 · 2025-05-03T06:31:23Z

Even if this is going to not be a default option, public instances owners will enable it because they want the best protection, and they will not understand the damages this is going to do.

There will be documentation that clearly describes what the disadvantages can be.

unixfox · 2025-05-03T20:15:33Z

I'm reiterating what I said, I'm not adding such feature. The built-in bot protection was created to reduce bots while fully knowing that anyone can bypass it by reading the source code, but I'm not turning it into a tool for restricting the access to a class of browsers family.

There are a myriad of other factors to catch bots, this PR is not the only ultimate factor to block bots or reduce the effects from bots:

http2: [botdetection] Require HTTP2 for major browsers #4709
ja4h: [botdetection] Implement JA4H #4710
captcha solver: Built-in CAPTCHA solver #2844
other ideas for blocking based on non-js usage: Support non JS users TecharoHQ/anubis#95

By the end of next week, I'm merging this PR as is, feel free to give me some feedback in case you test it.

return42

Improved version of #3965 which check for the browser version before checking the Sec-Fetch headers.

In my opinion, this PR is a weakening of the protection we got through PR #3965, if you think this weakening is necessary, then (rebase and) merge this PR.

…#4696)" This reverts commit 19b116f. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

…earxng#4696)" Many bots will probably use other user agents than the common ones (or rotate different ones)[1]. On my instance I can observe how bots with other UA headers are no longer stopped with the patch of PR searxng#4696. This reverts PR searxng#4696 commit 19b116f. [1] searxng#4696 (review) Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

return42 · 2025-05-20T12:11:09Z

I think that some bots use Firefox user agents, so this should be helpful for such.

I'm not sure if the general impact is as good as with @return42's PR because many bots will probably use other user agents than the common ones (or rotate different ones) - we'll have to see how it performs in production.

See my DRAFT:

[fix] Revert "fix: check if the browser supports Sec-Fetch headers (#4696)" #4807

…#4696)" This reverts commit 19b116f. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

…earxng#4696)" Many bots will probably use other user agents than the common ones (or rotate different ones)[1]. On my instance I can observe how bots with other UA headers are no longer stopped with the patch of PR searxng#4696. This reverts PR searxng#4696 commit 19b116f. [1] searxng#4696 (review) Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

unixfox requested a review from return42 April 30, 2025 09:14

unixfox mentioned this pull request Apr 30, 2025

[mod] botdetection: HTTP Fetch Metadata Request Headers #3965

Merged

unixfox force-pushed the Sec-Fetch-Mode-improved branch from deaf06b to ce2cb8b Compare April 30, 2025 09:19

unixfox changed the title ~~botdetection: HTTP Fetch Metadata Request Headers~~ [mod] botdetection: HTTP Fetch Metadata Request Headers - improved Apr 30, 2025

unixfox force-pushed the Sec-Fetch-Mode-improved branch from ce2cb8b to 399a31f Compare April 30, 2025 09:24

Bnyro approved these changes Apr 30, 2025

View reviewed changes

return42 reviewed May 4, 2025

View reviewed changes

fix: check if the browser supports Sec-Fetch headers

7373429

unixfox force-pushed the Sec-Fetch-Mode-improved branch from 399a31f to 7373429 Compare May 4, 2025 08:07

unixfox changed the title ~~[mod] botdetection: HTTP Fetch Metadata Request Headers - improved~~ fix: check if the browser supports Sec-Fetch headers May 4, 2025

unixfox changed the title ~~fix: check if the browser supports Sec-Fetch headers~~ [fix] botdetection: check if the browser supports Sec-Fetch headers May 4, 2025

unixfox merged commit 19b116f into master May 4, 2025
7 checks passed

return42 deleted the Sec-Fetch-Mode-improved branch May 4, 2025 10:47

return42 added a commit to return42/searxng that referenced this pull request May 20, 2025

Revert "fix: check if the browser supports Sec-Fetch headers (searxng…

0f7e3d7

…#4696)" This reverts commit 19b116f. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

return42 mentioned this pull request May 20, 2025

[fix] Revert "fix: check if the browser supports Sec-Fetch headers (#4696)" #4807

Draft

return42 added a commit to return42/searxng that referenced this pull request May 20, 2025

Revert "fix: check if the browser supports Sec-Fetch headers (searxng…

409bf5c

…#4696)" This reverts commit 19b116f. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

return42 added a commit to return42/searxng that referenced this pull request May 20, 2025

Revert "fix: check if the browser supports Sec-Fetch headers (searxng…

5621c56

…#4696)" This reverts commit 19b116f. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

return42 added a commit to return42/searxng that referenced this pull request May 23, 2025

Revert "fix: check if the browser supports Sec-Fetch headers (searxng…

3aba9c0

…#4696)" This reverts commit 19b116f. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

return42 added a commit to return42/searxng that referenced this pull request May 23, 2025

Revert "fix: check if the browser supports Sec-Fetch headers (searxng…

ecf6476

…#4696)" This reverts commit 19b116f. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

return42 added a commit to return42/searxng that referenced this pull request May 23, 2025

Revert "fix: check if the browser supports Sec-Fetch headers (searxng…

b13a45f

…#4696)" This reverts commit 19b116f. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

return42 added a commit to return42/searxng that referenced this pull request May 29, 2025

Revert "fix: check if the browser supports Sec-Fetch headers (searxng…

4917db8

…#4696)" This reverts commit 19b116f. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

return42 added a commit to return42/searxng that referenced this pull request Jun 3, 2025

Revert "fix: check if the browser supports Sec-Fetch headers (searxng…

167d23a

…#4696)" This reverts commit 19b116f. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

Bnyro pushed a commit to Bnyro/searxng that referenced this pull request Jun 25, 2025

fix: check if the browser supports Sec-Fetch headers (searxng#4696)

3623d3f

return42 added a commit to return42/searxng that referenced this pull request Jun 28, 2025

Revert "fix: check if the browser supports Sec-Fetch headers (searxng…

8dbb00e

…#4696)" This reverts commit 19b116f. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

[fix] botdetection: check if the browser supports Sec-Fetch headers #4696

[fix] botdetection: check if the browser supports Sec-Fetch headers #4696

Uh oh!

Conversation

unixfox commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Bnyro left a comment

Choose a reason for hiding this comment

Uh oh!

unixfox commented Apr 30, 2025

Uh oh!

unixfox commented May 2, 2025

Uh oh!

return42 commented May 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

unixfox commented May 2, 2025

Uh oh!

return42 commented May 2, 2025

Uh oh!

unixfox commented May 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

return42 commented May 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

unixfox commented May 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

return42 commented May 3, 2025

Uh oh!

unixfox commented May 3, 2025

Uh oh!

return42 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

return42 commented May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

unixfox commented Apr 30, 2025 •

edited

Loading

return42 commented May 2, 2025 •

edited

Loading

unixfox commented May 2, 2025 •

edited

Loading

return42 commented May 2, 2025 •

edited

Loading

unixfox commented May 2, 2025 •

edited

Loading

return42 left a comment •

edited

Loading

return42 commented May 20, 2025 •

edited

Loading