[question] hostname plugin: what results do you ignore? #970

return42 · 2022-01-07T15:49:25Z

return42
Jan 7, 2022
Maintainer

FYI: the old hostname_replace plugin has been superseded by the hostnames plugin:

[BREAKING CHANGE] hostname replace plugin rewrite #3463

I open this issue to share some experience about results (domains) that can be ignored.

For reference: collected lists

Updated 17/12/2024: Here is my current hostnames: setup:

hostnames:

  replace:
    # twitter --> nitter
    '(www\.)?twitter\.com$': 'nitter.net'

  remove:

    # block bload domains

    - 'com\.de'
    - '(.*\.)?askinglot\.com'
    - '(.*\.)?flipboard\.com'

    - '(.*\.)?codegrepper\.com'
    - '(.*\.)?w3schools\.com'
    - '(.*\.)?geeksforgeeks\.org'
    - '(.*\.)?stackshare\.io'
    - '(.*\.)?tutorialspoint\.com'
    - '(.*\.)?answeright\.com'
    - '(.*\.)?askdev\.info'
    - '(.*\.)?askdev\.io'
    - '(.*\.)?blogmepost\.com'
    - '(.*\.)?c-sharpcorner\.com'
    - '(.*\.)?code-examples\.net'
    - '(.*\.)?codeflow\.site'
    - '(.*\.)?gitmemory\.cn'
    - '(.*\.)?gitmemory\.com'
    - '(.*\.)?intellipaat\.com'
    - '(.*\.)?javaer101\.com'
    - '(.*\.)?programmerstart\.com'
    - '(.*\.)?programmersought\.com'
    - '(.*\.)?qastack\.com'
    - '(.*\.)?roboflow\.ai'
    - '(.*\.)?stackanswers\.net'
    - '(.*\.)?stackoom\.com'
    - '(.*\.)?stackovernet\.com'
    - '(.*\.)?stackovernet\.xyz'
    - '(.*\.)?stackoverrun\.com'
    - '(.*\.)?thetopsites\.net'
    - '(.*\.)?ubuntugeeks\.com'
    - '(.*\.)?cyberciti\.biz'
    - '(.*\.)?ispycode\.com'
    - '(.*\.)?reposhub\.com'
    - '(.*\.)?githubmemory\.com'
    - '(.*\.)?issueexplorer\.com'
    - '(.*\.)?tabnine\.com'
    - '(.*\.)?gitcode\.net'
    - '(.*\.)?command-not-found\.com'
    - '(.*\.)?im-coder\.com'
    - '(.*\.)?i-harness\.com'

    # ublock-origin-shitty-copies-filter/stackoverflow-copycats.txt

    - '(.*\.)?code-examples\.net'
    - '(.*\.)?kotaeta\.com'
    - '(.*\.)?answer-id\.com'
    - '(.*\.)?code\.i-harness\.com'
    - '(.*\.)?living-sun\.com'
    - '(.*\.)?qastack\.jp'
    - '(.*\.)?qastack\.ru'
    - '(.*\.)?qastack\.it'
    - '(.*\.)?qastack\.mx'
    - '(.*\.)?qastack\.com\.br'
    - '(.*\.)?qastack\.info\.tr'
    - '(.*\.)?qastack\.in\.th'
    - '(.*\.)?qastack\.com\.de'
    - '(.*\.)?qastack\.fr'
    - '(.*\.)?qastack\.cn'
    - '(.*\.)?qastack\.com\.ua'
    - '(.*\.)?qastack\.co\.in'
    - '(.*\.)?qastack\.kr'
    - '(.*\.)?qastack\.vn'
    - '(.*\.)?qastack\.net\.bd'
    - '(.*\.)?qa-stack\.pl'
    - '(.*\.)?qastack\.id'
    - '(.*\.)?coder\.work'
    - '(.*\.)?it-swarm-ja\.tech'
    - '(.*\.)?it-swarm\.jp\.net'
    - '(.*\.)?it-mure\.jp\.net'
    - '(.*\.)?it-swarm-ja\.com'
    - '(.*\.)?it-swarm\.com\.ru'
    - '(.*\.)?codeflow\.site'
    - '(.*\.)?codeguides\.site'
    - '(.*\.)?overcoder\.net'
    - '(.*\.)?coderoad\.ru'
    - '(.*\.)?generacodice\.blog'
    - '(.*\.)?generacodice\.it'
    - '(.*\.)?generacodice\.com'
    - '(.*\.)?javaer101\.com'
    - '(.*\.)?voidcc\.com'
    - '(.*\.)?siwib\.org'
    - '(.*\.)?fluffyfables\.com'
    - '(.*\.)?fixes\.pub'
    - '(.*\.)?knews\.vip'
    - '(.*\.)?isolution\.pro'
    - '(.*\.)?uwenku\.com'
    - '(.*\.)?lycaeum\.dev'
    - '(.*\.)?brocante\.dev'
    - '(.*\.)?classmethod\.dev'
    - '(.*\.)?stormcrow\.dev'
    - '(.*\.)?athabasca-foto\.com'
    - '(.*\.)?zsharp\.org'
    - '(.*\.)?projectbackpack\.org'
    - '(.*\.)?waymanamechurch\.org'
    - '(.*\.)?sunflowercreations\.org'
    - '(.*\.)?cfadnc\.org'
    - '(.*\.)?fitforlearning\.org'
    - '(.*\.)?panaindustrial\.com'
    - '(.*\.)?sierrasummit2005\.org'
    - '(.*\.)?theshuggahpies\.com'
    - '(.*\.)?pcbconline\.org'
    - '(.*\.)?nuomiphp\.com'
    - '(.*\.)?ubuntu\.buildwebhost\.com'
    - '(.*\.)?ubuntuaa\.com'
    - '(.*\.)?debugcn\.com'
    - '(.*\.)?sch22\.org'
    - '(.*\.)?gupgallery\.com'
    - '(.*\.)?amuddycup\.com'
    - '(.*\.)?ecnf2016\.org'
    - '(.*\.)?softwareuser\.asklobster\.com'
    - '(.*\.)?domainelespailles\.net'
    - '(.*\.)?ec-europe\.org'
    - '(.*\.)?pakostnici\.com'
    - '(.*\.)?try2explore\.com'
    - '(.*\.)?laravelquestions\.com'
    - '(.*\.)?itectec\.com'
    - '(.*\.)?stackovergo\.com'
    - '(.*\.)?faithcov\.org'
    - '(.*\.)?noblenaz\.org'
    - '(.*\.)?py4u\.net'
    - '(.*\.)?culinarydegree\.info'
    - '(.*\.)?safehavenpetrescue\.org'
    - '(.*\.)?qapicks\.com'
    - '(.*\.)?issues-world\.com'
    - '(.*\.)?narkive\.jp'
    - '(.*\.)?ourladylakes\.org'
    - '(.*\.)?intellipaat\.com'
    - '(.*\.)?article\.docway\.net'
    - '(.*\.)?xiu2\.net'
    - '(.*\.)?codehero\.jp'
    - '(.*\.)?quabr\.com'
    - '(.*\.)?webdevqa\.jp\.net'
    - '(.*\.)?qapicks\.com'
    - '(.*\.)?exceptionshub\.com'
    - '(.*\.)?newbedev\.com'
    - '(.*\.)?codenong\.com'
    - '(.*\.)?cainiaojiaocheng\.com'
    - '(.*\.)?routinepanic\.com'
    - '(.*\.)?pythonwd\.com'
    - '(.*\.)?tutorialmore\.com'
    - '(.*\.)?alwaysemmyhope\.com'
    - '(.*\.)?pretagteam\.com'
    - '(.*\.)?titanwolf\.org'
    - '(.*\.)?coderedirect\.com'
    - '(.*\.)?ostack\.cn'
    - '(.*\.)?easysavecode\.com'
    - '(.*\.)?webdevdesigner\.com'
    - '(.*\.)?ghcc\.net'
    - '(.*\.)?developreference\.com'
    - '(.*\.)?semicolonworld\.com'
    - '(.*\.)?tipsfordev\.com'
    - '(.*\.)?qi-u\.com'
    - '(.*\.)?stackqna\.com'
    - '(.*\.)?xsprogram\.com'
    - '(.*\.)?stackoom\.com'
    - '(.*\.)?it-swarm-fr\.com'
    - '(.*\.)?imtqy\.com'
    - '(.*\.)?christfever\.in'
    - '(.*\.)?codersatellite\.com'
    - '(.*\.)?cndgn\.com'
    - '(.*\.)?coredump\.biz'
    - '(.*\.)?farath\.com'
    - '(.*\.)?devbugfix\.com'
    - '(.*\.)?tech\.wayne-chu\.com'
    - '(.*\.)?stackify\.dev'
    - '(.*\.)?webknox\.com'
    - '(.*\.)?stackfinder\.jp\.net'
    - '(.*\.)?coder-solution-jp\.com'
    - '(.*\.)?jpndev\.com'
    - '(.*\.)?uebu-kaihatsu\.jp\.net'
    - '(.*\.)?sqlite\.in'
    - '(.*\.)?stackguides\.com'
    - '(.*\.)?younggeeks\.in'
    - '(.*\.)?answerlib\.com'
    - '(.*\.)?edupro\.id'
    - '(.*\.)?python-stack\.de'
    - '(.*\.)?stackfinder\.ru'
    - '(.*\.)?stackfault\.net'
    - '(.*\.)?howtofix\.io'
    - '(.*\.)?buzzphp\.com'
    - '(.*\.)?desenv-web-rp\.com'
    - '(.*\.)?askdev\.vn'
    - '(.*\.)?quares\.ru'
    - '(.*\.)?examplefiles\.net'
    - '(.*\.)?codewdw\.com'
    - '(.*\.)?jpcodeqa\.com'
    - '(.*\.)?tutorialguruji\.com'

     # ublock-origin-shitty-copies-filter/github-copycats.txt

    - '(.*\.)?bleepcoder.com'
    - '(.*\.)?githubja.com'
    - '(.*\.)?gitmemory.com'
    - '(.*\.)?giters.com'
    - '(.*\.)?githubmemory.com'
    - '(.*\.)?wenyanet.com'
    - '(.*\.)?issueexplorer.com'
    - '(.*\.)?opensourcelibs.com'
    - '(.*\.)?awesomeopensource.com'
    - '(.*\.)?findbestopensource.com'
    - '(.*\.)?githubhelp.com'
    - '(.*\.)?pythonissues.com'
    - '(.*\.)?golangrepo.com'
    - '(.*\.)?gitmemory.cn'
    - '(.*\.)?githubmate.com'
    - '(.*\.)?reposhub.com'
    - '(.*\.)?gitanswer.com'
    - '(.*\.)?githubplus.com'
    - '(.*\.)?pythonrepo.com'
    - '(.*\.)?bestofcpp.com'
    - '(.*\.)?github.innominds.com'
    - '(.*\.)?www.higithub.com'

k2s · 2022-01-09T00:28:11Z

k2s
Jan 9, 2022

based on https://github.com/jhchabran/code-search-blacklist I generated this settings with simple nodejs code:

hostname_replace:
   '(.*\.)?codegrepper\.com': false
   '(.*\.)?w3schools\.com': false
   '(.*\.)?geeksforgeeks\.org': false
   '(.*\.)?stackshare\.io': false
   '(.*\.)?tutorialspoint\.com': false
   '(.*\.)?answeright\.com': false
   '(.*\.)?askdev\.info': false
   '(.*\.)?askdev\.io': false
   '(.*\.)?blogmepost\.com': false
   '(.*\.)?c-sharpcorner\.com': false
   '(.*\.)?code-examples\.net': false
   '(.*\.)?codeflow\.site': false
   '(.*\.)?gitmemory\.cn': false
   '(.*\.)?gitmemory\.com': false
   '(.*\.)?intellipaat\.com': false
   '(.*\.)?javaer101\.com': false
   '(.*\.)?programmerstart\.com': false
   '(.*\.)?programmersought\.com': false
   '(.*\.)?qastack\.com': false
   '(.*\.)?roboflow\.ai': false
   '(.*\.)?stackanswers\.net': false
   '(.*\.)?stackoom\.com': false
   '(.*\.)?stackovernet\.com': false
   '(.*\.)?stackovernet\.xyz': false
   '(.*\.)?stackoverrun\.com': false
   '(.*\.)?thetopsites\.net': false
   '(.*\.)?ubuntugeeks\.com': false
   '(.*\.)?cyberciti\.biz': false
   '(.*\.)?ispycode\.com': false
   '(.*\.)?reposhub\.com': false
   '(.*\.)?githubmemory\.com': false
   '(.*\.)?issueexplorer\.com': false
   '(.*\.)?tabnine\.com': false
   '(.*\.)?gitcode\.net': false
   '(.*\.)?command-not-found\.com': false
   '(.*\.)?im-coder\.com': false
   '(.*\.)?i-harness\.com': false

1 reply

peepo5 Mar 18, 2022

codegrepper is finally gone woohoo

catfluoride · 2022-01-13T16:11:58Z

catfluoride
Jan 13, 2022

'(.*\.)?amazon\.(.*\.)?'

Because I don't buy stuff on Amazon but Amazon's results pollute my searches results whenever I want to look for reviews on specialized websites.

0 replies

masterofobzene · 2022-07-02T18:27:52Z

masterofobzene
Jul 2, 2022

Just here to reference another idea to filter sites.
User configurable negative words that append to search words automatically.

#1258 (comment)

0 replies

user451421541757324 · 2023-04-23T13:00:13Z

user451421541757324
Apr 23, 2023

I found this list.

2 replies

masterofobzene Apr 23, 2023

Nice, but forget it, these guys are not interested. Unsubscribing from this conversation as I lost all interest on searx due to its bad performance, constant blockades by the search engines and the lack of commitment on the team to fix it or make it better (they are more interested in discussing the "skins" used by the engine. I switched to startpage. Good luck though.

return42 Apr 24, 2023
Maintainer Author

I found this list.

@user451421541757324 thanks for pointing out 👍

Nice, but forget it, these guys are not interested. Unsubscribing from this conversation as I lost all interest on searx due to its bad performance,

@masterofobzene not sure .. do you mix the searx project with the SearXNG project?

the lack of commitment on the team to fix it or make it better

Here at SearXNG we active fix bugs, take a look at our issue tracker https://github.com/searxng/searxng/issues?q=is%3Aissue+is%3Aclosed

due to its bad performance, constant blockades by the search engines a

The SearXNG community maintains a lot of well working instances:

https://searx.space/
and we monitor the uptime https://uptime.searxng.org/

Bnyro · 2023-09-28T09:06:07Z

Bnyro
Sep 28, 2023
Maintainer

I don't specifically block any URLs, I only redirect a bunch of them to alternative, FOSS frontends:

  '(.*\.)?youtube\.com$': 'piped.video'
  '(.*\.)?youtu\.be$': 'piped.video'
  '(.*\.)?stackoverflow\.com$': 'code.whatever.social'
  'translate\.google\.com$': 'libretranslate.com'
  '(.*\.)?twitter\.com$': 'nitter.net'
  '(.*\.)?x\.com$': 'nitter.net'
  '(.*\.)?wikipedia.org$': 'wikiless.whateveritworks.org'
  '(.*\.)?quora.com$': 'quetre.iket.me'
  '(.*\.)?imdb\.com$': 'libremdb.iket.me'
  '(.*\.)?medium\.com$': 'scribe.rip'
  '(.*\.)?fandom\.com$': 'breezewiki.com'
  '(.*\.)?genius\.com$': 'intellectual.inspirill.net'
  '(.*\.)?reddit\.com$': 'reddit.smnz.de'
  '(.*\.)?tiktok.com$': 'proxitok.pabloferreiro.es'
  '(.*\.)?imgur.com$': 'rimgo.luncar.icu'

0 replies

Sommerwiesel · 2024-04-14T09:31:23Z

Sommerwiesel
Apr 14, 2024

Would also add:
'(.*.)?fextralife.com': false

to the list imo.

Also AnonymousOverflow finaly supports all stack pages, so these lines might be interesting too (using my instance of AnonymousOverflow as an example):

  # AnonymousOverflow
  '(.*\.)?stackoverflow\.com$': 'soflow.nerdvpn.de/exchange/stackoverflow.com'
  '(.*\.)?superuser\.com$': 'soflow.nerdvpn.de/exchange/superuser.com'
  '(.*\.)?serverfault\.com$': 'soflow.nerdvpn.de/exchange/serverfault.com'
  '(.*\.)?mathoverflow\.net$': 'soflow.nerdvpn.de/exchange/mathoverflow.net'
  '(.*\.)?stackapps\.com$': 'soflow.nerdvpn.de/exchange/stackapps.com'
  '(.*\.)?askubuntu\.com$': 'soflow.nerdvpn.de/exchange/askubuntu.com'
  '(.*\.)?stackexchange\.com$': 'soflow.nerdvpn.de/exchange/stackexchange.com'

9 replies

extremelyonline Jun 16, 2024

Thank you for your messages and sorry for the noise. I too confirm that the rules work. To cover the many subdomains of Stackexchange, I resorted to listing all of them out. Please let me know if there's a better way to do this:

'^(www\.)?stackexchange\.com$': 'code.whatever.social/exchange/stackexchange.com'\

'^(.*\.)?genai.stackexchange\.com$': 'code.whatever.social/exchange/genai.stackexchange.com'
'^(.*\.)?langdev.stackexchange\.com$': 'code.whatever.social/exchange/langdev.stackexchange.com'
'^(.*\.)?solana.stackexchange\.com$': 'code.whatever.social/exchange/solana.stackexchange.com'

# and so on...

AnonymousOverflow-rewrite.txt
(Github doesn't allow .yml uploading, please rename to .yml or add indentation if needed)

Sommerwiesel Jun 16, 2024

@extremelyonline
Holy shit, there are so many stack pages?!? WTF, almost makes me want to add some of them to the remove-hostnames. That's almost domain spamming if you ask me...

Thanks for sharing :)

extremelyonline Jun 16, 2024

Perhaps it's easier to approach this issue on the AnonymousOverflow side. Breezewiki has the same problem, and the redirections are handled by the instance maintainers.

Austin-Olacsi Jun 17, 2024

We wouldn't need a list with all of the many subdomains, if we could just make a single rule that would work with all of the subdomains. i had made a PR which would allow for that, it was a mod for the old hostname_replace plugin and it fixed the issue with breezewiki as well as well as AnonymousOverflow: #3051

that's what we should make now for the new hostnames plugin.

Sommerwiesel Jun 17, 2024

@Austin-Olacsi Agreed, adding more functionality to the regex replacement would be so much better.

extremelyonline · 2024-06-16T03:30:41Z

extremelyonline
Jun 16, 2024

Here's my redirection for twitter.com and x.com. Please let me know if there's a better way!

    '(.*\.)?m.twitter\.com$': 'xcancel.com'
    '(.*\.)?mobile.twitter\.com$': 'xcancel.com'
    '^(about\www\.)?twitter\.com$': 'xcancel.com'
    '(.*\.)?m.x\.com$': 'xcancel.com'
    '(.*\.)?mobile.x\.com$': 'xcancel.com'
    '^(about\www\.)?x\.com$': 'xcancel.com'

(The problem with '(.*\.)?x\.com$': 'xcancel.com' is that it replaces abcx.com with abcxcancel.com)

2 replies

Bnyro Jun 16, 2024
Maintainer

You can use '^(.*\.)?x\.com$' :)

aqxa1 Nov 24, 2024

You can use '^(.*\.)?x\.com$' :)

This doesn't work for me. Instead of the rule ignoring abcx.com, it wrongly redirects to xcancel.com. I am using the NixOS service, so perhaps there is an issue there.

This simpler rule does work:

'^x.com$': 'xcancel.com';

but doesn't handle subdomains. xcancel.com doesn't handle them in a useful way anyway AFAICT.

return42 · 2024-06-16T13:44:13Z

return42
Jun 16, 2024
Maintainer Author

Holy shit, there are so many stack pages?!? WTF, almost makes me want to add some of them to the remove-hostnames. That's almost domain spamming if you ask me...

What do you think about hosting lists in a repository under the SearXNG umbrella? I could imagine that the SearXNG community would benefit if we were to offer the know-how that we have already accumulated in part here in the discussion in an organized form in a repository.

elated: [FEATURE REQUEST] external list file for hostname_replace #2619 (comment)

1 reply

Sommerwiesel Jun 16, 2024

I was already thinking about that, too. Both of my replace and remove lists are heavily influenced by my knowledge of alternative frontends and my private pihole (my searxng remove list already features a lot of fake/seo scam pages that I took via python script from my pihole).

I currently don't have much time but I would totally volunteer creating, maintaining or contributing to such a repo in the near future.

Rushmore75 · 2024-12-16T22:13:38Z

Rushmore75
Dec 16, 2024

@return42 Do you have your list publicly available? That we could wget it or similar?

4 replies

return42 Dec 17, 2024
Maintainer Author

I updated the list in the first comment .. but please note, my list from above is far from being adequate .. it would better we had one list under the SearXNG umbrella as suggest here #970 (comment)

Sommerwiesel Dec 17, 2024

I updated the list in the first comment .. but please note, my list from above is far from being adequate .. it would better we had one list under the SearXNG umbrella as suggest here #970 (comment)

I would be willing to create and manage such a list under the searXNG umbrella. Have been using my own list for half a year now. Send me a PM on Matrix @Weidenwiesel:nerdvpn.de if you're interested and we'll talk about the details etc.

return42 Dec 17, 2024
Maintainer Author

I would be willing to create and manage such a list under the SearXNG umbrella.

@Sommerwiesel that sounds good 🚀 .. would you like to send me a mail markus.heiser@darmarit.de .. thanks!!

return42 Jan 30, 2025
Maintainer Author

@Sommerwiesel have you reached my email response? .. no need to hurry, I ask just to confirm my mail is not in your spam folder ;-)

xlionjuan · 2025-02-05T08:53:19Z

xlionjuan
Feb 5, 2025

What is "bload domains" means?

3 replies

Sommerwiesel Feb 5, 2025

In a nutshell how I see it: Domains/websites that are either smartly named or named like their original counterparts so that people mistakenly visit them.
Some of them then host stolen content as their own, but most of them host malicious content and scripts (for example JavaScript coin miners) or use shady redirects to actual bad sites etc etc so they should be avoided like the plague.

xlionjuan Feb 5, 2025

Flipboard
w3schools?

return42 Feb 5, 2025
Maintainer Author

List above is old .. feel welcome to make your suggestions.

return42 · 2025-04-23T09:00:49Z

return42
Apr 23, 2025
Maintainer Author

FYI: found well maintained DNS blocklists in the mullvad project:

https://github.com/mullvad/dns-blocklists/tree/main

0 replies

[question] hostname plugin: what results do you ignore? #970

Uh oh!

Uh oh!

return42 Jan 7, 2022 Maintainer

Replies: 11 comments · 22 replies

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

return42 Apr 24, 2023 Maintainer Author

Uh oh!

Bnyro Sep 28, 2023 Maintainer

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Bnyro Jun 16, 2024 Maintainer

Uh oh!

Uh oh!

return42 Jun 16, 2024 Maintainer Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

return42 Dec 17, 2024 Maintainer Author

Uh oh!

Uh oh!

Uh oh!

return42 Dec 17, 2024 Maintainer Author

Uh oh!

Uh oh!

return42 Jan 30, 2025 Maintainer Author

Uh oh!

return42
Jan 7, 2022
Maintainer

Replies: 11 comments 22 replies

return42 Apr 24, 2023
Maintainer Author

Bnyro
Sep 28, 2023
Maintainer

Bnyro Jun 16, 2024
Maintainer

return42
Jun 16, 2024
Maintainer Author

return42 Dec 17, 2024
Maintainer Author

return42 Dec 17, 2024
Maintainer Author

return42 Jan 30, 2025
Maintainer Author