[mod] isolation of botdetection from the limiter #2894
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
efe8f8d to
d8e906c
Compare
Contributor
|
|
Bnyro
reviewed
Oct 27, 2023
Bnyro
approved these changes
Oct 27, 2023
d8e906c to
5eb3f57
Compare
This patch was inspired by the discussion around PR-2882 [2]. The goals of this
patch are:
1. Convert plugin searx.plugin.limiter to normal code [1]
2. isolation of botdetection from the limiter [2]
3. searx/{tools => botdetection}/config.py and drop searx.tools
4. in URL /config, 'limiter.enabled' is true only if the limiter is really
enabled (Redis is available).
This patch moves all the code that belongs to botdetection into namespace
searx.botdetection and code that belongs to limiter is placed in namespace
searx.limiter.
Tthe limiter used to be a plugin at some point botdetection was added, it was
not a plugin. The modularization of these two components was long overdue.
With the clear modularization, the documentation could then also be organized
according to the architecture.
[1] searxng#2882
[2] searxng#2882 (comment)
To test:
- check the app works without the limiter, check `/config`
- check the app works with the limiter and with the token, check `/config`
- make docs.live .. and read
- http://0.0.0.0:8000/admin/searx.limiter.html
- http://0.0.0.0:8000/src/searx.botdetection.html#botdetection
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
…tion - the option server:public_instance lacks some documentation - the processing of this option belongs in the limiter and not in botdetection module Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
5eb3f57 to
76b41d3
Compare
dalf
reviewed
Nov 1, 2023
| cfg = get_cfg() | ||
| if settings['server']['public_instance']: | ||
| # overwrite limiter.toml setting | ||
| cfg.set('botdetection.ip_limit.link_token', True) |
Contributor
There was a problem hiding this comment.
I was thinking about this either at the top of this function or in settings_default.py:
if settings['server']['public_instance']:
# overwrite limiter.toml setting
cfg = get_cfg()
cfg.set('botdetection.ip_limit.link_token', True)
# overwrite settings.yml
settings['server']['limiter'] = Trueso:
- b05a155 can be revert
- line 222 can check
settings['server']['limiter']and there is no need to think aboutserver.public_instanceanymore in the future. - the code above describes all the changes made by
server.public_instancein one place.
Member
Author
There was a problem hiding this comment.
b05a155 can be revert
not yet, see the "or" ..
get_setting('server.limiter') or get_setting('server.public_instance')
the point is, that server.public_instance does not activate the server.limiter
I wanted to prepare a PR for this, but first I needed to merge this PR / to prevent a conflict of the new PR and this PR (the new PR needs to be on top of this PR here).
Contributor
There was a problem hiding this comment.
Contributor
There was a problem hiding this comment.
This new parameter for the moment activate by default both the bot limiter AND the new advanced bot limiter called link_token.
So yes it should activate server.limiter
return42
added a commit
to return42/searxng
that referenced
this pull request
Nov 1, 2023
In PR-2894[1] we isolated botdetection from the limiter, this PR isolates the botdetection from the SearXNG core code. This PR also fixes the issue [2] that the ``server.public_instance`` option needs to activate the limiter. - [1] searxng#2894 - [2] searxng#2975 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
return42
added a commit
to return42/searxng
that referenced
this pull request
Nov 1, 2023
In PR-2894[1] we isolated botdetection from the limiter, this PR isolates the botdetection from the SearXNG core code. This PR also fixes the issue [2] that the ``server.public_instance`` option needs to activate the limiter. - [1] searxng#2894 - [2] searxng#2975 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
return42
added a commit
to return42/searxng
that referenced
this pull request
Jan 9, 2024
In PR-2894[1] we isolated botdetection from the limiter, this PR isolates the botdetection from the SearXNG core code. This PR also fixes the issue [2] that the ``server.public_instance`` option needs to activate the limiter. - [1] searxng#2894 - [2] searxng#2975 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
return42
added a commit
to return42/searxng
that referenced
this pull request
Feb 25, 2024
In PR-2894[1] we isolated botdetection from the limiter, this PR isolates the botdetection from the SearXNG core code. This PR also fixes the issue [2] that the ``server.public_instance`` option needs to activate the limiter. - [1] searxng#2894 - [2] searxng#2975 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
return42
added a commit
to return42/searxng
that referenced
this pull request
Feb 26, 2024
In PR-2894[1] we isolated botdetection from the limiter, this PR isolates the botdetection from the SearXNG core code. This PR also fixes the issue [2] that the ``server.public_instance`` option needs to activate the limiter. - [1] searxng#2894 - [2] searxng#2975 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is an alternative to #2882
It mainly consist of two commits ...
[mod] isolation of botdetection from the limiterThis patch was inspired by the discussion around PR-2882 [2]. The goals of this
patch are:
enabled (Redis is available).
This patch moves all the code that belongs to botdetection into namespace
searx.botdetection and code that belongs to limiter is placed in namespace
searx.limiter.
Tthe limiter used to be a plugin at some point botdetection was added, it was
not a plugin. The modularization of these two components was long overdue.
With the clear modularization, the documentation could then also be organized
according to the architecture.
[1] #2882
[2] #2882 (comment)
[mod] document server:public_instance & remove it out of the botdetectionin botdetection module
To test:
/config/config