Convert plugin searx.plugin.limiter to normal code #2882

dalf · 2023-09-29T10:36:38Z

What does this PR do?

The limiter plugin is not a plugin:

the user can't enable or disable the plugin and there is no point to allow that
the limiter does not use any of the features provides by the plugin framework

This commit convert the limiter plugin into normal code

Additional change: in /config, limiter.enabled is true only if the limiter is really enabled (Redis is available).

Why is this change important?

The limiter is shown as a plugin here and there and this is confusing.

How to test this PR locally?

check the app works without the limiter, check /config
check the app works with the limiter and with the token, check /config

Author's checklist

The initialization code is in searx/botdetection/install.py, it can be in searx/botdetection/__init__.py in the same way that searx.search and searx.redisdb.
Alternative : move the code from install.py to limiter.py, but this lead to the next topic: naming ambiguity .

There are two names to reference the same thing:

limiter
searx.botdetection

Is there a difference between the botdetection and the limiter ?

Since limiter is the known public name, I suggest:

to rename searx.botdetection into searx.limiter
to move the code from searx/botdetection/limiter.py and searx/botdetection/install.py to searx/limiter/__init__.py.

Related issues

[EDIT] sorry for the numerous edits.

The limiter plugin is not a plugin: * the user can't enable or disable the plugin and there is no point to allow that * the limiter does use any of the features provides by the plugin framework This commit convert the limiter plugin into normal code

return42

Haven't tested .. but left some comments in the code to make it a little bit more readable .. what do you think?

searx/botdetection/install.py

searx/webapp.py

searx/botdetection/install.py

return42 · 2023-09-29T11:12:35Z

Additional change: in /config, limiter.enabled is true only if the limiter is really enabled (Redis is available).

You now, that the limiter can be enabled without redis? .. the methods that probe HTTP headers do not need a redis.

dalf · 2023-09-29T11:16:54Z

(Redis is available)

This is misleading, the variable is True only when the limiter is actually installed (Redis installed or not).

searxng/searx/botdetection/install.py

Lines 29 to 43 in bee313c

    
           def initialize(app: flask.Flask, settings): 
        
               """Instal the botlimiter aka limiter""" 
        
               global _INSTALLED  # pylint: disable=global-statement 
        
               if not settings['server']['limiter'] and not settings['server']['public_instance']: 
        
                   return 
        
               if not redisdb.client(): 
        
                   logger.error( 
        
                       "The limiter requires Redis, please consult the documentation: " 
        
                       + "https://docs.searxng.org/admin/searx.botdetection.html#limiter" 
        
                   ) 
        
                   if settings['server']['public_instance']: 
        
                       sys.exit(1) 
        
                   return 
        
               app.before_request(pre_request) 
        
               _INSTALLED = True

You can see that _INSTALLED = True is done at the end after the hook is installed.

dalf · 2023-09-29T12:48:26Z

For information the commit c5baf9b applied on top of this PR move searx.botdetection to searx.limiter.
@return42 if you think that make sense, I can add this commit to this PR.

return42 · 2023-09-29T12:53:23Z

if you think that make sense, I can add this commit to this PR.

not sure .. there are also some title or chapter naming that might be better changed from "botdetection" to "limiter" .. lets focus on the code in this PR .. and in a follow up we can discuss how to reorder the documentation.

return42 · 2023-09-30T08:43:49Z

There are two names to reference the same thing:

limiter
searx.botdetection

Is there a difference between the botdetection and the limiter ?

In the past we had said the limiter is a simple builtin that limits request from an IP. My intention was real bot-detection as an (optional) addition to the builtin limiter.

The searx.botdetection namespace has not much (none except redis?) dependency to the rest of the SearXNG core .. As far as possible, I also don't want to bury bot detection too deep in the SearXNG code.

Bot detection is a task that has nothing to do with the core of SearXNG, this should also be clearly visible on namespace level. The "module" botdetection should in principle be interchangeable and the limiter is the integration in SearXNG.

The modularization of limiter and botdetection should keep the possibility open for us to integrate external modules or components into the limiter in the future.

The only thing that would be missing at the moment is the definition of an interface between the limiter and the (external) bot detection. The currently existing modularization in the code would make this very easy. If we now merge the limiter with the botdetection this will not be so easy in the future.

TL;DR; just imagine that today's namespace searx.botdetection is an external module that you can install.

--> this is also the reason why this module has its own configuration ..

🤔 OK we can discuss if limiter.toml was a well chosen name :)

dalf · 2023-09-30T09:59:09Z

So the name is botdetection not limiter.
However, everything outside this module is named limiter: the configuration options, configuration file, the API, etc...

There is no need to solve this in this PR: here, the purpose is just to remove the plugin [*] does not make sense.

[*] plugin as they are defined now.

This patch was inspired by the discussion around PR-2882 [2]. The goals of this patch are: 1. Convert plugin searx.plugin.limiter to normal code [1] 2. isolation of botdetection from the limiter [2] This patch moves all the code that belongs to botdetection into namespace searx.botdetection and code that belongs to limiter is placed in namespace searx.limiter. Tthe limiter used to be a plugin at some point botdetection was added, it was not a plugin. The modularization of these two components was long overdue. With the clear modularization, the documentation could then also be organized according to the architecture. [1] searxng#2882 [2] searxng#2882 (comment) Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

This patch was inspired by the discussion around PR-2882 [2]. The goals of this patch are: 1. Convert plugin searx.plugin.limiter to normal code [1] 2. isolation of botdetection from the limiter [2] 3. searx/{tools => botdetection}/config.py and drop searx.tools 4. in URL /config, 'limiter.enabled' is true only if the limiter is really enabled (Redis is available). This patch moves all the code that belongs to botdetection into namespace searx.botdetection and code that belongs to limiter is placed in namespace searx.limiter. Tthe limiter used to be a plugin at some point botdetection was added, it was not a plugin. The modularization of these two components was long overdue. With the clear modularization, the documentation could then also be organized according to the architecture. [1] searxng#2882 [2] searxng#2882 (comment) To test: - check the app works without the limiter, check `/config` - check the app works with the limiter and with the token, check `/config` - make docs.live .. and read - http://0.0.0.0:8000/admin/searx.limiter.html - http://0.0.0.0:8000/src/searx.botdetection.html#botdetection Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

dalf · 2023-10-03T04:31:19Z

Supersede by #2894

This patch was inspired by the discussion around PR-2882 [2]. The goals of this patch are: 1. Convert plugin searx.plugin.limiter to normal code [1] 2. isolation of botdetection from the limiter [2] 3. searx/{tools => botdetection}/config.py and drop searx.tools 4. in URL /config, 'limiter.enabled' is true only if the limiter is really enabled (Redis is available). This patch moves all the code that belongs to botdetection into namespace searx.botdetection and code that belongs to limiter is placed in namespace searx.limiter. Tthe limiter used to be a plugin at some point botdetection was added, it was not a plugin. The modularization of these two components was long overdue. With the clear modularization, the documentation could then also be organized according to the architecture. [1] searxng#2882 [2] searxng#2882 (comment) To test: - check the app works without the limiter, check `/config` - check the app works with the limiter and with the token, check `/config` - make docs.live .. and read - http://0.0.0.0:8000/admin/searx.limiter.html - http://0.0.0.0:8000/src/searx.botdetection.html#botdetection Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

This patch was inspired by the discussion around PR-2882 [2]. The goals of this patch are: 1. Convert plugin searx.plugin.limiter to normal code [1] 2. isolation of botdetection from the limiter [2] 3. searx/{tools => botdetection}/config.py and drop searx.tools 4. in URL /config, 'limiter.enabled' is true only if the limiter is really enabled (Redis is available). This patch moves all the code that belongs to botdetection into namespace searx.botdetection and code that belongs to limiter is placed in namespace searx.limiter. Tthe limiter used to be a plugin at some point botdetection was added, it was not a plugin. The modularization of these two components was long overdue. With the clear modularization, the documentation could then also be organized according to the architecture. [1] #2882 [2] #2882 (comment) To test: - check the app works without the limiter, check `/config` - check the app works with the limiter and with the token, check `/config` - make docs.live .. and read - http://0.0.0.0:8000/admin/searx.limiter.html - http://0.0.0.0:8000/src/searx.botdetection.html#botdetection Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

dalf requested review from return42 and unixfox September 29, 2023 10:36

dalf force-pushed the limiter_as_normal_code branch from c5b0d86 to bee313c Compare September 29, 2023 10:37

return42 requested changes Sep 29, 2023

View reviewed changes

searx/botdetection/install.py Show resolved Hide resolved

searx/webapp.py Show resolved Hide resolved

searx/webapp.py Show resolved Hide resolved

searx/botdetection/install.py Show resolved Hide resolved

return42 mentioned this pull request Oct 2, 2023

[mod] isolation of botdetection from the limiter #2894

Merged

dalf closed this Oct 3, 2023

dalf deleted the limiter_as_normal_code branch October 3, 2023 04:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Convert plugin searx.plugin.limiter to normal code #2882

Convert plugin searx.plugin.limiter to normal code #2882

Uh oh!

dalf commented Sep 29, 2023 •

edited

Loading

Uh oh!

return42 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

return42 commented Sep 29, 2023

Uh oh!

dalf commented Sep 29, 2023

Uh oh!

dalf commented Sep 29, 2023

Uh oh!

return42 commented Sep 29, 2023

Uh oh!

return42 commented Sep 30, 2023

Uh oh!

dalf commented Sep 30, 2023 •

edited

Loading

Uh oh!

dalf commented Oct 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Convert plugin searx.plugin.limiter to normal code #2882

Convert plugin searx.plugin.limiter to normal code #2882

Uh oh!

Conversation

dalf commented Sep 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is this change important?

How to test this PR locally?

Author's checklist

Related issues

Uh oh!

return42 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

return42 commented Sep 29, 2023

Uh oh!

dalf commented Sep 29, 2023

Uh oh!

dalf commented Sep 29, 2023

Uh oh!

return42 commented Sep 29, 2023

Uh oh!

return42 commented Sep 30, 2023

Uh oh!

dalf commented Sep 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dalf commented Oct 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dalf commented Sep 29, 2023 •

edited

Loading

dalf commented Sep 30, 2023 •

edited

Loading