Where is the automatic language detection code? #2669

mehr32 · 2023-08-23T10:17:23Z

mehr32
Aug 23, 2023

Hello, where is the path of the automatic search language detection code (in the air filter, not the settings)? And how does it work?

Answered by return42

Aug 23, 2023

Its in the util.py ...

searxng/searx/utils.py

Lines 555 to 623 in 4b42644

     def detect_language(text: str, threshold: float = 0.3, only_search_languages: bool = False) -> Optional[str]:  
   """Detect the language of the ``text`` parameter.  
     
    :param str text: The string whose language is to be detected.  
     
    :param float threshold: Threshold filters the returned labels by a threshold  
    on probability. A choice of 0.3 will return labels with at least 0.3  
    probability.  
     
    :param bool only_search_languages: If ``True``, returns only supported  
    SearXNG search languages. see :py:obj:`searx.languages`  
     
    :rtype: str, None  
    :returns:  

View full answer

return42 · 2023-08-23T10:22:01Z

return42
Aug 23, 2023
Maintainer

Its in the util.py ...

searxng/searx/utils.py

Lines 555 to 623 in 4b42644

    
           def detect_language(text: str, threshold: float = 0.3, only_search_languages: bool = False) -> Optional[str]: 
        
               """Detect the language of the ``text`` parameter. 
        
               :param str text: The string whose language is to be detected. 
        
               :param float threshold: Threshold filters the returned labels by a threshold 
        
                   on probability.  A choice of 0.3 will return labels with at least 0.3 
        
                   probability. 
        
               :param bool only_search_languages: If ``True``, returns only supported 
        
                   SearXNG search languages.  see :py:obj:`searx.languages` 
        
               :rtype: str, None 
        
               :returns: 
        
                   The detected language code or ``None``. See below. 
        
               :raises ValueError: If ``text`` is not a string. 
        
               The language detection is done by using `a fork`_ of the fastText_ library 
        
               (`python fasttext`_). fastText_ distributes the `language identification 
        
               model`_, for reference: 
        
               - `FastText.zip: Compressing text classification models`_ 
        
               - `Bag of Tricks for Efficient Text Classification`_ 
        
               The `language identification model`_ support the language codes 
        
               (ISO-639-3):: 
        
                   af als am an ar arz as ast av az azb ba bar bcl be bg bh bn bo bpy br bs 
        
                   bxr ca cbk ce ceb ckb co cs cv cy da de diq dsb dty dv el eml en eo es 
        
                   et eu fa fi fr frr fy ga gd gl gn gom gu gv he hi hif hr hsb ht hu hy ia 
        
                   id ie ilo io is it ja jbo jv ka kk km kn ko krc ku kv kw ky la lb lez li 
        
                   lmo lo lrc lt lv mai mg mhr min mk ml mn mr mrj ms mt mwl my myv mzn nah 
        
                   nap nds ne new nl nn no oc or os pa pam pfl pl pms pnb ps pt qu rm ro ru 
        
                   rue sa sah sc scn sco sd sh si sk sl so sq sr su sv sw ta te tg th tk tl 
        
                   tr tt tyv ug uk ur uz vec vep vi vls vo wa war wuu xal xmf yi yo yue zh 
        
               By using ``only_search_languages=True`` the `language identification model`_ 
        
               is harmonized with the SearXNG's language (locale) model.  General 
        
               conditions of SearXNG's locale model are: 
        
               a. SearXNG's locale of a query is passed to the 
        
                  :py:obj:`searx.locales.get_engine_locale` to get a language and/or region 
        
                  code that is used by an engine. 
        
               b. Most of SearXNG's engines do not support all the languages from `language 
        
                  identification model`_ and there is also a discrepancy in the ISO-639-3 
        
                  (fastext) and ISO-639-2 (SearXNG)handling.  Further more, in SearXNG the 
        
                  locales like ``zh-TH`` (``zh-CN``) are mapped to ``zh_Hant`` 
        
                  (``zh_Hans``) while the `language identification model`_ reduce both to 
        
                  ``zh``. 
        
               .. _a fork: https://github.com/searxng/fasttext-predict 
        
               .. _fastText: https://fasttext.cc/ 
        
               .. _python fasttext: https://pypi.org/project/fasttext/ 
        
               .. _language identification model: https://fasttext.cc/docs/en/language-identification.html 
        
               .. _Bag of Tricks for Efficient Text Classification: https://arxiv.org/abs/1607.01759 
        
               .. _`FastText.zip: Compressing text classification models`: https://arxiv.org/abs/1612.03651 
        
               """ 
        
               if not isinstance(text, str): 
        
                   raise ValueError('text must a str') 
        
               r = _get_fasttext_model().predict(text.replace('\n', ' '), k=1, threshold=threshold) 
        
               if isinstance(r, tuple) and len(r) == 2 and len(r[0]) > 0 and len(r[1]) > 0: 
        
                   language = r[0][0].split('__label__')[1] 
        
                   if only_search_languages and language not in SEARCH_LANGUAGE_CODES: 
        
                       return None 
        
                   return language 
        
               return None

0 replies

mehr32 · 2023-08-23T10:30:00Z

mehr32
Aug 23, 2023
Author

thank you . Sir, does he also use the language of the browser? Because most of the time, it does not recognize the language of the searched text correctly and returns only the language of the browser. What is the solution? (I have a searxng sample)

0 replies

return42 · 2023-08-23T10:37:44Z

return42
Aug 23, 2023
Maintainer

There is a threshold

searxng/searx/webadapter.py

Lines 262 to 263 in 4b42644

    
           if query_lang == 'auto': 
        
               query_lang = detect_language(query, threshold=0.8, only_search_languages=True)

if lang detection does not reach the threshold, browser's language is used:

searxng/searx/webapp.py

Lines 192 to 195 in 4b42644

    
           def _get_browser_language(req, lang_list): 
        
               client = ClientPref.from_http_request(req) 
        
               locale = match_locale(client.locale_tag, lang_list, fallback='en') 
        
               return locale

0 replies

mehr32 · 2023-08-23T10:40:58Z

mehr32
Aug 23, 2023
Author

Well, this lowers the accuracy of the work and consequently the quality of the results. Why don't you use more accurate libraries for language recognition or online services?

0 replies

return42 · 2023-08-23T10:57:03Z

return42
Aug 23, 2023
Maintainer

Well, this lowers the accuracy of the work and consequently the quality of the results.

lowers by what and compared to what? ..

Why don't you use more accurate libraries

You know a better lib? .. (that does not to much memory) .. we optimized fasttext to lower the footprint --> https://github.com/searxng/fasttext-predict

or online services?

Would add a HTTP request/response round-trip (on server side) to each search a user sends .. the response of the result list will take longer / we try to minimize the search response to as little as possible. Most time consumption is in the HTTP request/response cycles / OT: that is why we also optimized such round-trips in the CSS & JS ..

0 replies

mehr32 · 2023-08-23T11:28:22Z

mehr32
Aug 23, 2023
Author

In the tests that I did, this method does not work well on non-English languages, especially with short text lengths, and most of the time, the browser language is used, so the results are diverted to the browser language.

This Python library has worked well in my use: langdetect

That's right, online service is not good for this

11 replies

mehr32 Aug 23, 2023
Author

----> Failed to connect: Potential security issue
When the browser language is Farsi:
search : hi : fa
helo : fa
Is Linux a beautiful operating system? : fa ! :/

Now, if it is in English, I will test it.

mehr32 Aug 23, 2023
Author

that is weird ! If the browser is in English:
سلام : en
ایا لینوکس سیستم عاملی زیباست ؟ : fa !

dalf Aug 23, 2023

Disclaimer : this comment does not describe a solution, it is just one finding.

On huggingface, there is a model which seems provide support for Persian language:

https://huggingface.co/facebook/fasttext-language-identification

You can try it directly in the browser: on the right side bellow "Hosted inference API / Text Classification".

However:

the license is showstopper
one comment says Inaccurate language detection for short phrases like greetings
the model is 150MB instead of 1MB. It can be download here (see this PR, this is the quantized version of model, the original model is 1GB).
the results are different : eng_Latn instead of __label__en for example: SearXNG can load the model but detect_language can't use it without code change. See this other PR to get the list of possible results

Perhaps there is smaller model somewhere else, or we can use another library according to the alphabet of the query.

[EDIT] There is another model with a GPL3 license, but still 150MB on disk, 1GB in RAM.

mehr32 Aug 24, 2023
Author

I really don't know! As you said, langdetect did not work well in practice. Thank you in any case. I was happy to meet you!

return42 Aug 24, 2023
Maintainer

Thank you in any case. I was happy to meet you!

feel welcome / we need your help .. can you please help to translate SearXNG's UI:

Persian: https://translate.codeberg.org/translate/searxng/searxng/fa/?q=state:%3Ctranslated

You can make suggestion as anonymous / no need to create a login if you don't want to.

	def detect_language(text: str, threshold: float = 0.3, only_search_languages: bool = False) -> Optional[str]:
	"""Detect the language of the ``text`` parameter.

	:param str text: The string whose language is to be detected.

	:param float threshold: Threshold filters the returned labels by a threshold
	on probability. A choice of 0.3 will return labels with at least 0.3
	probability.

	:param bool only_search_languages: If ``True``, returns only supported
	SearXNG search languages. see :py:obj:`searx.languages`

	:rtype: str, None
	:returns:

Where is the automatic language detection code? #2669

Uh oh!

mehr32 Aug 23, 2023

Replies: 6 comments · 11 replies

Uh oh!

return42 Aug 23, 2023 Maintainer

Uh oh!

mehr32 Aug 23, 2023 Author

Uh oh!

return42 Aug 23, 2023 Maintainer

Uh oh!

mehr32 Aug 23, 2023 Author

Uh oh!

return42 Aug 23, 2023 Maintainer

Uh oh!

mehr32 Aug 23, 2023 Author

Uh oh!

mehr32 Aug 23, 2023 Author

Uh oh!

mehr32 Aug 23, 2023 Author

Uh oh!

Uh oh!

dalf Aug 23, 2023

Uh oh!

mehr32 Aug 24, 2023 Author

Uh oh!

return42 Aug 24, 2023 Maintainer

mehr32
Aug 23, 2023

Replies: 6 comments 11 replies

return42
Aug 23, 2023
Maintainer

mehr32
Aug 23, 2023
Author

return42
Aug 23, 2023
Maintainer

mehr32
Aug 23, 2023
Author

return42
Aug 23, 2023
Maintainer

mehr32
Aug 23, 2023
Author

mehr32 Aug 23, 2023
Author

mehr32 Aug 23, 2023
Author

mehr32 Aug 24, 2023
Author

return42 Aug 24, 2023
Maintainer