这是indexloc提供的服务，不要输入任何密码
Skip to content

Fix update engine traits (brave & annas archive) #2853

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Sep 27, 2023
Merged

Fix update engine traits (brave & annas archive) #2853

merged 3 commits into from
Sep 27, 2023

Conversation

@return42
Copy link
Member

@return42 return42 commented Sep 26, 2023

What does this PR do?

fix the fetch traits function of:

  • brave
  • annas archive

Why is this change important?

As @dalf pointed out, the gh workflow to fetch the traits from the engines did fails since month --> #2826 (comment)

How to test this PR locally?

make data.traits

Related issues

@return42
[fix] engine annas archive - fetch traits (modified xpath selectors)
b67f36d 
Anna’s Archive has cleaned up their languages, available file extensions and
changed the HTML form.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
@return42
[fix] engine brave - fetch traits (modified settings menu)
d689f52 
Brave has changed it settings menu fundamental.  Region codes are no longer in
the HTML page, we have to read the regional codes from a JS:

  https://cdn.search.brave.com/serp/v2/_app/immutable/chunks/parameters.734c106a.js

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
@return42
Update searx.data - update_engine_traits.py
3c3ce81 
Update searx.data - update_engine_traits.py

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
@return42 return42 requested review from Bnyro and dalf September 26, 2023 11:14
Bnyro
Bnyro reviewed Sep 26, 2023
View reviewed changes
searx/engines/brave.py
# search regions of brave

engine_traits.all_locale = 'all'
resp = get('https://cdn.search.brave.com/serp/v2/_app/immutable/chunks/parameters.734c106a.js', headers=headers)
Copy link
Member

@Bnyro Bnyro Sep 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could probably break really quick again, but I don't have a better idea unfortunately

Copy link
Member Author

@return42 return42 Sep 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I have checked: the URL is not a href in a <link> of the root html from https://search.brave.com/settings .. this doc loads some JS and one of the loaded JS scripts extends the <link> list in the root document to load the parameters.734c106a.js' .. very complicated. Instead of having two hard to track indirectness I decided to read directly from parameters.734c106a.js .. and see when it breaks :o

@return42 return42 mentioned this pull request Sep 26, 2023
Merged
@return42
Copy link
Member Author

return42 commented Sep 27, 2023 •
edited
Loading

@dalf on my local instance make data.traits did work / do you have a tip for me, how I can test in CI? .. I triggered the job in the CI https://github.com/return42/searxng/actions/runs/6322049251 but the job was skipped.

If it is to complicated, can we merge and test in SearXNG's master?

@dalf
Copy link
Contributor

dalf commented Sep 27, 2023 •
edited
Loading

You need to add a temporary commit to remove this line:
https://github.com/return42/searxng/blob/3c3ce81f314224bd31647ac01a417f07562c924f/.github/workflows/data-update.yml#L11

Since there is no way to debug the GitHub actions, prepare yourself to git add .; git commit --amend; git push --force, and once okay clean the git log of your branch.

If it is to complicated, can we merge and test in SearXNG's master?

It will create PR in searxng/searxng : if there are broken, it will be a mess. It is better to test in a fork.

dalf
dalf approved these changes Sep 27, 2023
View reviewed changes
Copy link
Contributor

@dalf dalf left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It runs fine in your fork.
Once it runs fine in your fork, I let you remove the WIP commit and self merge.

@return42 return42 force-pushed the fix-update_engine_traits branch from 2e34a3c to 3c3ce81 Compare September 27, 2023 06:56
@return42
Copy link
Member Author

return42 commented Sep 27, 2023 •
edited
Loading

You need to add a temporary commit to remove this line ..

Yeah, that did the trick .. PR was created in my clone: return42#254

It runs fine in your fork.

But I also observed, what I have also seen local on my desktop, 502 Bad Gateway from zlibrary-global.se:

raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Server error '520 ' for url 'https://zlibrary-global.se/'

Seems to me zlibrary-global.se is sometimes unstable .. since this issue is not related to this PR, I'm going to merge this PR.

@return42 return42 merged commit 0c39064 into searxng:master Sep 27, 2023
@return42 return42 deleted the fix-update_engine_traits branch September 27, 2023 07:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Bnyro Bnyro Bnyro approved these changes

+1 more reviewer

@dalf dalf dalf approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@return42 @dalf @Bnyro