Apply black #619

dalf · 2021-12-19T19:02:44Z

What does this PR do?

This PR is demo of black:

first commit black -l 120 -S searx searxng_extra : reformat except the simple quote.
second commit : black -l 120 searx searxng_extra: reformat including the simple quote.

Why is this change important?

Normalize code format.

How to test this PR locally?

Author's checklist

About PEP8 E203: see Does not comply with E203 from pep8 psf/black#315
If we use black, we can disable pycodestyle, and just reformat the code (and keep pylint). The tooling is not modified, so make test fails for now.
there is an emacs package: https://github.com/wbolster/emacs-python-black

Related issues

return42 · 2021-12-21T05:56:45Z

I started to implement https://github.com/return42/searxng/commits/black-code-style
/ but I struggle when adding a pyproject.toml file.

A pyproject.toml breaks our python installation procedure with a misleading error message ... I don't know why users of "black" are forced to change their build stack just to configure a tool .. I gave up.

Here is the error message, pyyaml is installed in the virtualenv, but with a pyproject.toml, pip reports that a yaml installation is missed .. very confusing.

$ make buildenv
...
BUILDENV  PYENV     [install] pip install -e 'searx[test]'
BUILDENV  Obtaining file:///x/share/searxng
BUILDENV    Installing build dependencies: started
BUILDENV    Installing build dependencies: finished with status 'done'
BUILDENV    Checking if build backend supports build_editable: started
BUILDENV    Checking if build backend supports build_editable: finished with status 'done'
BUILDENV    Getting requirements to build wheel: started
BUILDENV    Getting requirements to build wheel: finished with status 'error'
BUILDENV    ERROR: Command errored out with exit status 1:
BUILDENV     command: /x/share/searxng/local/py3/bin/python /x/share/searxng/local/py3/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpkffhqb23
BUILDENV         cwd: /x/share/searxng
BUILDENV    Complete output (22 lines):
BUILDENV    Traceback (most recent call last):
BUILDENV      File "/x/share/searxng/local/py3/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
BUILDENV        main()
BUILDENV      File "/x/share/searxng/local/py3/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
BUILDENV        json_out['return_val'] = hook(**hook_input['kwargs'])
BUILDENV      File "/x/share/searxng/local/py3/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
BUILDENV        return hook(config_settings)
BUILDENV      File "/tmp/pip-build-env-g39jipud/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 162, in get_requires_for_build_wheel
BUILDENV        return self._get_build_requires(
BUILDENV      File "/tmp/pip-build-env-g39jipud/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 143, in _get_build_requires
BUILDENV        self.run_setup()
BUILDENV      File "/tmp/pip-build-env-g39jipud/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 267, in run_setup
BUILDENV        super(_BuildMetaLegacyBackend,
BUILDENV      File "/tmp/pip-build-env-g39jipud/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 158, in run_setup
BUILDENV        exec(compile(code, __file__, 'exec'), locals())
BUILDENV      File "setup.py", line 7, in <module>
BUILDENV        from searx.version import VERSION_TAG, GIT_URL
BUILDENV      File "/x/share/searxng/searx/__init__.py", line 12, in <module>
BUILDENV        import searx.settings_loader
BUILDENV      File "/x/share/searxng/searx/settings_loader.py", line 8, in <module>
BUILDENV        import yaml
BUILDENV    ModuleNotFoundError: No module named 'yaml'
BUILDENV    ----------------------------------------
BUILDENV  WARNING: Discarding file:///x/share/searxng. Command errored out with exit status 1: /x/share/searxng/local/py3/bin/python /x/share/searxng/local/py3/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpkffhqb23 Check the logs for full command output.

dalf · 2021-12-21T10:32:44Z

pyaml is required to run setup.py, so it has be declared as such in project.toml:

But to make it simple why not run black with parameters:
black -l 120 --target-version py36
?

black is not very configurable, but as described, it would solve the code formatting discussion.

return42 · 2021-12-21T10:38:16Z

black is not very configurable, but as described, it would solve the code formatting discussion.

Personally I prefer a config file .. what can be read by other tools also. I don't know if we should reinvent our "requirements" setup to have such a toml file .. ?

not-my-profile · 2021-12-22T11:28:34Z

How about using --skip-string-normalization? I don't think that string normalization adds much ... rather it needlessly produces a huge diff obscuring git blame ... (yes I know you can use git blame --skip-rev but normally you only use that after seeing that the autoformat commit changed the line you are interested in). So I think --skip-string-normalization would improve git blame ergonomics.

That is how about omitting the second commit?

dalf · 2021-12-24T09:58:00Z

I pushed a new version:

A first commit update ./manage:

make test.pep8 is replaced by test.black: black runs on ALL python files including linted by pylint.
make format.python calls black to format the Python code

A second commit contains the result of make format.python

A third one fixes some pylint issues (it can be improved, but at least the tests pass).

return42 · 2021-12-24T10:33:08Z

I pushed a new version

fine, thanks .. I will have a look.

What do you think about pyproject.toml in our project, should I give it a try? If so I would take time for a research (its new to me).

[EDIT] now I see there is a --config option / no need to have a pyproject.toml ..

return42 · 2021-12-24T11:17:57Z

searx/search/checker/impl.py

 HTML_TAGS = [
-    'embed', 'iframe', 'object', 'param', 'picture', 'source', 'svg', 'math', 'canvas', 'noscript', 'script',
-    'del', 'ins', 'area', 'audio', 'img', 'map', 'track', 'video', 'a', 'abbr', 'b', 'bdi', 'bdo', 'br', 'cite',
-    'code', 'data', 'dfn', 'em', 'i', 'kdb', 'mark', 'q', 'rb', 'rp', 'rt', 'rtc', 'ruby', 's', 'samp', 'small',
-    'span', 'strong', 'sub', 'sup', 'time', 'u', 'var', 'wbr', 'style', 'blockquote', 'dd', 'div', 'dl', 'dt',
-    'figcaption', 'figure', 'hr', 'li', 'ol', 'p', 'pre', 'ul', 'button', 'datalist', 'fieldset', 'form', 'input',
-    'label', 'legend', 'meter', 'optgroup', 'option', 'output', 'progress', 'select', 'textarea', 'applet',
-    'frame', 'frameset'
+    'embed',
+    'iframe',
+    'object',
+    'param',
+    'picture',
+    'source',
+    'svg',
+    'math',
+    'canvas',
+    'noscript',
+    'script',
+    'del',
+    'ins',
+    'area',
+    'audio',
+    'img',
+    'map',
+    'track',
+    'video',
+    'a',
+    'abbr',
+    'b',
+    'bdi',
+    'bdo',
+    'br',
+    'cite',
+    'code',
+    'data',
+    'dfn',
+    'em',
+    'i',
+    'kdb',
+    'mark',
+    'q',
+    'rb',
+    'rp',
+    'rt',
+    'rtc',
+    'ruby',
+    's',
+    'samp',
+    'small',
+    'span',
+    'strong',
+    'sub',
+    'sup',
+    'time',
+    'u',
+    'var',
+    'wbr',
+    'style',
+    'blockquote',
+    'dd',
+    'div',
+    'dl',
+    'dt',
+    'figcaption',
+    'figure',
+    'hr',
+    'li',
+    'ol',
+    'p',
+    'pre',
+    'ul',
+    'button',
+    'datalist',
+    'fieldset',
+    'form',
+    'input',
+    'label',
+    'legend',
+    'meter',
+    'optgroup',
+    'option',
+    'output',
+    'progress',
+    'select',
+    'textarea',
+    'applet',
+    'frame',
+    'frameset',
 ]


Over all I think black formatting is good, but there are some exceptions like this one here.

In such cases we should use # fmt: off & # fmt: on.

return42 · 2021-12-24T11:26:24Z

FYI: I am in review with this PR, I will push some suggestions, please do not change the code while I am in review / thanks!

return42

LGTM / thanks!

@dalf See my comment and the patches I pushed on top of your branch.

return42 · 2021-12-25T09:26:06Z

We should prioritize (merge) this PR over:

@dalf if you want I can rebase this branch, but first have a look at the commits I pushed on top of your branch.

dalf · 2021-12-27T07:16:54Z

first have a look at the commits I pushed on top of your branch.

c699eb8 : okay, why not.
1480058 : 👍

if you want I can rebase this branch

it will be never a good time to change a lot of time files. So let's go now.

return42 · 2021-12-27T07:22:27Z

it will be never a good time to change a lot of time files. So let's go now.

OK, I will going to rebase this branch right now, please do not merge other PR with python code in the meantime ..

"make test.black" checks for the code style "make format.python" format the python code

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

Disable the python code formatting from python-black, where the readability of code suffers by formatting. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

This patch was generated by black [1]:: make format.python [1] https://github.com/psf/black Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

Fix pylint issues from commit (3d96a98) [format.python] initial formatting of the python code Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

return42 · 2021-12-27T09:34:48Z

@dalf I rebased and reorder the branch / If you think PR is ready to merge, please merge.

FYI: the CI hangs for some reason I don't know, I restarted the jobs .. lets see if the tests pass now ...

return42 · 2021-12-27T09:46:10Z

Seems ./manage node.env does not work in the CI (see here) / not related to this PR I think.

dalf · 2021-12-27T18:11:27Z

Thank you for the update.

return42 mentioned this pull request Dec 20, 2021

[pylint] Bing (Videos) engine #614

Merged

not-my-profile mentioned this pull request Dec 21, 2021

[doc] introduce about.language and sort engines by it #629

Merged

dalf force-pushed the apply-black branch from 005d80e to 4e1f3f8 Compare December 24, 2021 09:52

return42 reviewed Dec 24, 2021

View reviewed changes

return42 approved these changes Dec 24, 2021

View reviewed changes

return42 marked this pull request as ready for review December 24, 2021 13:02

dalf and others added 5 commits December 27, 2021 08:58

[mod] replace pycodestyle by black

3c77412

"make test.black" checks for the code style "make format.python" format the python code

[mod] test.black: show issues (diff) reported by python-black

c849731

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

[format.python] disable py code formatting for some hunks of code

fcdc2c2

Disable the python code formatting from python-black, where the readability of code suffers by formatting. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

[format.python] initial formatting of the python code

3d96a98

This patch was generated by black [1]:: make format.python [1] https://github.com/psf/black Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

[fix] issues reported by pylint

d84226b

Fix pylint issues from commit (3d96a98) [format.python] initial formatting of the python code Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

return42 force-pushed the apply-black branch from 1480058 to d84226b Compare December 27, 2021 09:19

dalf merged commit c6922ae into searxng:master Dec 27, 2021

dalf deleted the apply-black branch December 27, 2021 18:11

return42 mentioned this pull request Aug 5, 2022

[fix] pycodestyle has been replaced by black in 3c77412d3 #1618

Merged

return42 mentioned this pull request Sep 18, 2025

[mod] addition of various type hints / engine processors #5204

Merged

Apply black #619

Apply black #619

Uh oh!

Conversation

dalf commented Dec 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is this change important?

How to test this PR locally?

Author's checklist

Related issues

Uh oh!

return42 commented Dec 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dalf commented Dec 21, 2021

Uh oh!

return42 commented Dec 21, 2021

Uh oh!

not-my-profile commented Dec 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dalf commented Dec 24, 2021

Uh oh!

return42 commented Dec 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

return42 Dec 24, 2021

Choose a reason for hiding this comment

Uh oh!

return42 commented Dec 24, 2021

Uh oh!

return42 left a comment

Choose a reason for hiding this comment

Uh oh!

return42 commented Dec 25, 2021

Uh oh!

dalf commented Dec 27, 2021

Uh oh!

return42 commented Dec 27, 2021

Uh oh!

return42 commented Dec 27, 2021

Uh oh!

return42 commented Dec 27, 2021

Uh oh!

dalf commented Dec 27, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dalf commented Dec 19, 2021 •

edited

Loading

return42 commented Dec 21, 2021 •

edited

Loading

not-my-profile commented Dec 22, 2021 •

edited

Loading

return42 commented Dec 24, 2021 •

edited

Loading