Update __init__.py #15396
Update __init__.py #15396
Conversation
Added code to detect (via the chardet module) file type encoding, then pass the detected encoding to the appropriate '.read_text' function. Also added 'log.info' statements to state which files are being read along with their detected file character encoding
|
We require contributors to sign our Contributor License Agreement and we don't have one on file for @Rich-AM. In order for us to review and merge your code, please e-sign the Contributor License Agreement PDF. We then need to manually verify your signature, merge the PR (conda/infrastructure#1238), and ping the bot to refresh the PR. |
|
json must be utf-8 only. It might be worth checking that conda works correctly under less-common default encodings. https://peps.python.org/pep-0686/ makes utf-8 default in Python 3.15 |
|
@conda-bot check |
|
Some additional context on this from ChatGPT: On most Windows systems, the default character encoding used by Python is However, this can vary depending on the system locale settings.
So, in short:
With the above in mind and what you have described, I think you bring up some valid points here, but I don't think we'll accept the solution as you proposed because it's not as efficient as it could be (you load the entire file into memory and these files are normally many megabytes if not hundreds), and these errors aren't very widespread because most If you would like to see better character encoding in conda, I suggest opening a separate issue as a "feature" request to clearly definitely the problem and how we can solve it. Also, for future pull requests to this repository, it's very important to have tests that validate the solution you have proposed. If you are ever unsure of how to write these tests or where to place them please reach out to us via these pull requests or our chat/message board: https://conda.zulipchat.com |
Added code to detect (via the chardet module) file type encoding, then pass the detected encoding to the appropriate '.read_text' function. Also added 'log.info' statements to state which files are being read along with their detected file character encoding
Description
Reason for the change.
If a conda channel's
repodata.jsonfile contains any non-ASCII (e.g. UTF-8) characters, conda defaults to the local workstation regional file encoding. For Linux and Mac, this is typically UTF-8, for Windows, it is the system regional default (cp-1252 Western Latin for most US based Windows systems). If a single conda channel contains a UTF-8 encodedrepodata.jsonfile, it can cause a Windows installed version of conda to fail due to an 'invalid character' being detected. Example: If the unicode character U+201D (the right double quote character) is in therepodata.jsonfile for a channel added to the Windows machine conda environment, conda will fail even a simple command such asconda search pandasThe changes proposed in this
__init__.pyfile merely uses thechardetmodule to detect the file's character encoding, then passes the detected encoding to the appropriate.read_textfunction. Additionallylog.infostatements have been added to provide users with additional insight into which files are being read (cached or downloaded) as well as their detected file character encoding.Lastly, all changes were bracketed by a string of 20 '#' with specific comments added.
Thank you for time and consideration of this request
Checklist - did you ...
I did not do any of the following checklist items as I did not think this change request was significant enough
[ ] Add a file to thenewsdirectory (using the template) for the next release's release notes?[ ] Add / update necessary tests?[ ] Add / update outdated documentation?