'content': ' | '.join(content) #4632

dalf · 2025-04-14T19:37:28Z

dalf
Apr 14, 2025

Various engines don't have a proper content, but rather a list of key / value;

The content ends up to be something like this : ' | '.join(content)

Would it make sense to:

either update the default.html template to allow key / value
or add another template based on default.html to replace the content by a list of key / value.

Engines:

quark (at least according to the code)
presearch (at least according to the code)

brave:

searxng/searx/engines/brave.py

Lines 329 to 335 in 3033061

    
           content: str = extract_text( 
        
               eval_xpath_getindex(result, './/div[contains(@class, "snippet-description")]', 0, default='') 
        
           )  # type: ignore 
        
           pub_date_raw = eval_xpath(result, 'substring-before(.//div[contains(@class, "snippet-description")], "-")') 
        
           pub_date = _extract_published_date(pub_date_raw) 
        
           if pub_date and content.startswith(pub_date_raw): 
        
               content = content.lstrip(pub_date_raw).strip("- \n\t")

chefkoch:

searxng/searx/engines/chefkoch.py

Lines 49 to 53 in 3033061

    
           content = [ 
        
               f"Schwierigkeitsstufe (1-3): {recipe['difficulty']}", 
        
               f"Zubereitungszeit: {recipe['preparationTime']}min", 
        
               f"Anzahl der Zutaten: {recipe['ingredientCount']}", 
        
           ]

deezer:

searxng/searx/engines/deezer.py

Line 53 in 3033061

content = '{} - {} - {}'.format(result['artist']['name'], result['album']['title'], result['title'])

fdroid:

searxng/searx/engines/fdroid.py

Lines 45 to 49 in 3033061

    
           app_content = ( 
        
               extract_text(app.xpath('./div/div/span[@class="package-summary"]')).strip() 
        
               + ' - ' 
        
               + extract_text(app.xpath('./div/div/span[@class="package-license"]')).strip() 
        
           )

geizhals:

searxng/searx/engines/geizhals.py

Lines 72 to 74 in 3033061

    
           content = [] 
        
           for spec in eval_xpath_list(result, ".//div[contains(@class, 'specs-grid__item')]"): 
        
               content.append(f"{extract_text(eval_xpath(spec, './dt'))}: {extract_text(eval_xpath(spec, './dd'))}")

gitea:

searxng/searx/engines/gitea.py

Line 96 in 3033061

content = [item.get(i) for i in ['language', 'description'] if item.get(i)]
github:

searxng/searx/engines/github.py

Line 39 in 3033061

content = [item.get(i) for i in ['language', 'description'] if item.get(i)]

google_news:

searxng/searx/engines/google_news.py

Lines 157 to 162 in 3033061

    
           # The pub_date is mostly a string like 'yesterday', not a real 
        
           # timezone date or time.  Therefore we can't use publishedDate. 
        
           pub_date = extract_text(eval_xpath(result, './article//time')) 
        
           pub_origin = extract_text(eval_xpath(result, './article//a[@data-n-tid]')) 
        
           content = ' / '.join([x for x in [pub_origin, pub_date] if x])

huggingface:

searxng/searx/engines/huggingface.py

Lines 98 to 106 in 3033061

    
           contents = [] 
        
           if entry.get("likes"): 
        
               contents.append(f"Likes: {entry['likes']}") 
        
           if entry.get("downloads"): 
        
               contents.append(f"Downloads: {entry['downloads']:,}") 
        
           if entry.get("tags"): 
        
               contents.append(f"Tags: {', '.join(entry['tags'])}") 
        
           if entry.get("description"): 
        
               contents.append(f"Description: {entry['description']}")

imdb:

searxng/searx/engines/imdb.py

Lines 65 to 71 in 3033061

    
           content = '' 
        
           if 'rank' in entry: 
        
               content += "(%s) " % entry['rank'] 
        
           if 'y' in entry: 
        
               content += str(entry['y']) + " - " 
        
           if 's' in entry: 
        
               content += entry['s']

livespace:

searxng/searx/engines/livespace.py

Lines 65 to 67 in 3033061

    
           content = [] 
        
           if category: 
        
               content.append(f'category - {category}')

loc:

searxng/searx/engines/loc.py

Lines 74 to 79 in 3033061

    
           content_items = [ 
        
               result['item'].get('created_published_date'), 
        
               result['item'].get('summary', [None])[0], 
        
               result['item'].get('notes', [None])[0], 
        
               result['item'].get('part_of', [None])[0], 
        
           ]

material_icons:

searxng/searx/engines/material_icons.py

Line 57 in 3033061

'content': ", ".join(tags) + " / " + ", ".join(categories),
moviepilot:

searxng/searx/engines/moviepilot.py

Line 110 in 3033061

item['content'] = ', '.join([result['class'], result['info'], result['more']])

mrs:

searxng/searx/engines/mrs.py

Lines 62 to 65 in 3033061

    
           'content': result['topic'] 
        
           + f" // {result['members']} members" 
        
           + f" // {result['alias']}" 
        
           + f" // {result['server']}",

nyaa:

searxng/searx/engines/nyaa.py

Lines 103 to 105 in 3033061

    
           # content string contains all information not included into template 
        
           content = 'Category: "{category}". Downloaded {downloads} times.' 
        
           content = content.format(category=category, downloads=downloads)

pdbe:

searxng/searx/engines/pdbe.py

Lines 104 to 106 in 3033061

    
           content = '{msg_superseded}: {url} ({pdb_id})'.format( 
        
               msg_superseded=msg_superseded, url=superseded_url, pdb_id=result['superseded_by'] 
        
           )

public_domain_image_archive:

searxng/searx/engines/public_domain_image_archive.py

Lines 127 to 134 in 3033061

    
           content = [] 
        
           if "themes" in result: 
        
               content.append("Themes: " + result['themes']) 
        
           if "encompassingWork" in result: 
        
               content.append("Encompassing work: " + result['encompassingWork']) 
        
           content = "\n".join(content)

qwant:

searxng/searx/engines/qwant.py

Lines 288 to 294 in 3033061

    
           if d: 
        
               content_parts.append(d) 
        
           if s: 
        
               content_parts.append("%s: %s " % (gettext("Source"), s)) 
        
           if c: 
        
               content_parts.append("%s: %s " % (gettext("Channel"), c)) 
        
           content = ' // '.join(content_parts)

radio_browser:

searxng/searx/engines/radio_browser.py

Lines 104 to 112 in 3033061

    
           content = [] 
        
           tags = ', '.join(result.get('tags', '').split(',')) 
        
           if tags: 
        
               content.append(tags) 
        
           for x in ['state', 'country']: 
        
               v = result.get(x) 
        
               if v: 
        
                   v = str(v).strip() 
        
                   content.append(v)

rottentomatoes:

searxng/searx/engines/rottentomatoes.py

Lines 42 to 48 in 3033061

    
           content = [] 
        
           for xpath in (release_year_xpath, score_xpath, cast_xpath): 
        
               info = extract_text(eval_xpath(result, xpath)) 
        
               # a gap in the end means that no data was found 
        
               if info and info[-1] != " ": 
        
                   content.append(info)

rumble:

searxng/searx/engines/rumble.py

Lines 69 to 72 in 3033061

    
           if earned: 
        
               content = f"{views} views - {rumbles} rumbles - ${earned}" 
        
           else: 
        
               content = f"{views} views - {rumbles} rumbles"

senscritique:

searxng/searx/engines/senscritique.py

Lines 114 to 144 in b61ef7a

    
           def build_content_parts(item: dict[str, Any], title: str, original_title: Optional[str]) -> list[str]: 
        
               """Build the content parts for an item""" 
        
               content_parts = [] 
        
               if item.get('category'): 
        
                   content_parts.append(item['category']) 
        
               if original_title and original_title != title: 
        
                   content_parts.append(f"Original title: {original_title}") 
        
               if item.get('directors'): 
        
                   directors = [director['name'] for director in item['directors']] 
        
                   content_parts.append(f"Director(s): {', '.join(directors)}") 
        
               if item.get('countries'): 
        
                   countries = [country['name'] for country in item['countries']] 
        
                   content_parts.append(f"Country: {', '.join(countries)}") 
        
               if item.get('genresInfos'): 
        
                   genres = [genre['label'] for genre in item['genresInfos']] 
        
                   content_parts.append(f"Genre(s): {', '.join(genres)}") 
        
               if item.get('duration'): 
        
                   minutes = item['duration'] // 60 
        
                   if minutes > 0: 
        
                       content_parts.append(f"Duration: {minutes} min") 
        
               if item.get('rating') and item.get('stats', {}).get('ratingCount'): 
        
                   content_parts.append(f"Rating: {item['rating']}/10 ({item['stats']['ratingCount']} votes)") 
        
               return content_parts

soundcloud:

searxng/searx/engines/soundcloud.py

Lines 82 to 85 in 3033061

    
           content = [ 
        
               result.get("description"), 
        
               result.get("label_name"), 
        
           ]

spotify:

searxng/searx/engines/spotify.py

Line 64 in 3033061

content = '{} - {} - {}'.format(result['artists'][0]['name'], result['album']['name'], result['name'])

stackexchange:

searxng/searx/engines/stackexchange.py

Lines 56 to 60 in 3033061

    
           content = "[%s]" % ", ".join(result['tags']) 
        
           content += " %s" % result['owner']['display_name'] 
        
           if result['is_answered']: 
        
               content += ' // is answered' 
        
           content += " // score: %s" % result['score']

steam:

searxng/searx/engines/steam.py

Line 40 in 3033061

content = [f'Price: {price:.2f} {currency}', f'Platforms: {platforms}']
voidlinux:

searxng/searx/engines/voidlinux.py

Line 68 in 3033061

'content': f"{result['short_desc']} - {humanize_bytes(result['filename_size'])}",
yandex_music:

searxng/searx/engines/yandex_music.py

Line 58 in 3033061

'content': f"[{result['albums'][0]['title']}] {result['artists'][0]['name']} - {result['title']}",

~~I most probably miss some other engines.~~

[EDIT] message edited to add the engines concat multiple fields in the content

return42 · 2025-04-15T05:32:38Z

return42
Apr 15, 2025
Maintainer

Various engines don't have a proper content, but rather a list of key / value;

🤔 can't say I often missed an option for key/value pairs in the content .. but often I have seen lists when implementing a result of an engine.

Would it make sense to:

either update the default.html template to allow key / value

or add another template based on default.html to replace the content by a list of key / value.

What about a third alternative: in the rendering (default.html) offer paragraph (what we have now) and additional key/value pairs and list types.

Meanwhile we have typed results for the main result list. For most templates we have not yet implemented a type (class), but at least for the default.html template there is already a class MainResult.

In its current state MainResult.content has to be a string .. we can patch the content property to accept str, list and dict (ordered?) .. haven't thought to its end but I would suggest we can render results (content block) like the following one that consist of a paragraph, a list and a list of key value pairs:

Lorem ipsum dolor sit amet, consetetur sadipscing elitr,
sed diam nonumy eirmod tempor invidunt ut labore et dolore
magna aliquyam erat, sed diam voluptua.

- first list item
- second list item 

- foo: value of foo
- bar: value of bar

The data structure of the content above would be a list of str, list and dict:

content = [
    "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua.",
    ["first list item", "second list item"],
    {"foo": "value of foo", "bar": "value of bar"},
]

@dalf does this fit to what you have in mind?

0 replies

dalf · 2025-04-15T15:28:00Z

dalf
Apr 15, 2025
Author

What about a third alternative: in the rendering (default.html) offer paragraph (what we have now) and additional key/value pairs and list types.

Yes this

content = [
"Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua.",
["first list item", "second list item"],
{"foo": "value of foo", "bar": "value of bar"},
]

I think about something like this

WhateverClassResult(
  content = "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua.",
  properties= {"foo": "value of foo", "bar": "value of bar"},  # of keyvalues
)

so there is no need to deal with a complex value for content: it is remains a string nothing more.

0 replies

dalf · 2025-04-26T19:51:03Z

dalf
Apr 26, 2025
Author

I've updated my original message to list the various engines that concat multiple fields for the content.

Common fields

Here some common fields that can be extracted from all these concated contents

Programming language

packages.html
- gitea (language)
- github (language)

Tag list and category list

general.html
- huggingface (tag)
- radio_browser (tag)
- senscritique (one category, genre)
videos.html
- livespace (category)
images.html
- loc (part_of)
- material_icons (tag & category)

License

general.html
- fdroid (XPath package-license)
- huggingface (lost in the tags, but in clear in the JSON response, one tag has the prefix license:)
packages.html
- github / gitlab / gitea (not shown, but I expect this field in the JSON response)

Engagement metrics

general.html
- huggingface (like, downloads)
- rottentomatoes (score)
- rumble (views, rumbles) <-- note: the scrapping of these values is broken.
- imdb (rank) <--- I had to look at the source code to understand what was the number
- senscritique (rating)

Published date, most of the time not parsable

general.html
- google_news
- brave (pub_date)
images.html
- loc (created_published_date)
- rottentomatoes (release_year)

Location

general.html
- senscritique (country)
- radio_browser (state, country)
- huggingface (there is a tag region:xx)

Fields seen once, not repeated elsewhere

Specific to one engine but it make sense for the whole SearXNG category:

general.html
- senscritique
  - directors
  - original_title
  - duration
- deezer
  - album title
  - artist name
- moviepilot
  - media type (film, series, etc...)
- rottentomatoes
  - cast
- chefkoch
  - recipe difficulty
  - recipe preparation time
  - ingredient count
- geizhals
  - price (actually the engine allows any key & value but I see only one key "Bestes Angebot" / Best offer)

From that some suggestions (this require more thinking, just a draft) :

create a media.html template (film / series / video)
- title
- original_title
- media_type
- duration
- release_year
- location
- rating / views / score
create a music.html template
- title
- artist
- album
- duration
- release_year
- rating / views / score
create a recipe.html template
create a apps.html template
- add a license field
packages.html:
- add a language field
- add a license field
default.html / videos.html / images.html (all ?)
- allow a list of tags and categories

This is quiet different from allowing random key/value, but I think it would make the results more readable.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

'content': ' | '.join(content) #4632

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 3 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

'content': ' | '.join(content) #4632

Uh oh!

Uh oh!

dalf Apr 14, 2025

Replies: 3 comments

Uh oh!

return42 Apr 15, 2025 Maintainer

Uh oh!

dalf Apr 15, 2025 Author

Uh oh!

Uh oh!

dalf Apr 26, 2025 Author

Common fields

Programming language

Tag list and category list

License

Engagement metrics

Published date, most of the time not parsable

Location

Fields seen once, not repeated elsewhere

dalf
Apr 14, 2025

return42
Apr 15, 2025
Maintainer

dalf
Apr 15, 2025
Author

dalf
Apr 26, 2025
Author