Conda 4.7+ extremely slow "Collecting package metadata (repodata.json)" with local (file:///) channels #9661
Description
When installing or updating packages, conda gets stuck for an incredibly long time at the "Collecting package metadata (repodata.json)" step.
This appears to be a regression caused by conda 4.7 (and remains in 4.8) as this operation was quite fast in conda 4.6.
The slowdown only appears to occur when using (file:///) channels on a local or network filesystem.
To test this I ran python -m http.server 8000 in the channel parent directory and renamed channel URLs in .condarc from file:///<DRIVE>:/<PATH> to http://localhost:8000/. Commands then run in around the same time as they did in conda 4.6.
Steps to Reproduce
conda update conda or any other conda update/install command
conda-debug.txt
(note I exited (Ctrl+c) the command after a few minutes, normally it would run for an hour or more)
DEBUG conda.gateways.logging:set_verbosity(231): verbosity set to 2
DEBUG conda.core.solve:solve_final_state(223): solving prefix C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3
specs_to_remove: frozenset()
specs_to_add: frozenset({MatchSpec("conda")})
prune: <auxlib._Null object at 0x0000024AA6792148>
DEBUG conda.core.package_cache_data:_check_writable(259): package cache directory 'C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3\pkgs' writable: True
DEBUG conda.core.subdir_data:_load(241): Local cache timed out for file:///X:/<LOCAL-MIRROR-PATH>/msys2/noarch/repodata.json at C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3\pkgs\cache\e8bdfbda.json
DEBUG conda.core.subdir_data:_load(241): Local cache timed out for file:///X:/<LOCAL-MIRROR-PATH>/msys2/win-64/repodata.json at C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3\pkgs\cache\270cd9d6.json
DEBUG conda.core.subdir_data:_load(241): Local cache timed out for file:///X:/<LOCAL-MIRROR-PATH>/main/win-64/repodata.json at C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3\pkgs\cache\a83aae50.json
DEBUG conda.core.subdir_data:_load(241): Local cache timed out for file:///X:/<LOCAL-MIRROR-PATH>/main/noarch/repodata.json at C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3\pkgs\cache\ce3ccb3a.json
INFO conda._vendor.auxlib.logz:stringify(144): request is 'None' for Response object with url file:///X:/<LOCAL-MIRROR-PATH>/msys2/noarch/repodata.json
DEBUG conda.core.subdir_data:fetch_repodata_remote_request(487):
<<FILE 200 None
< Content-Type: application/json
< Last-Modified: Tue, 13 Nov 2018 03:00:11 GMT
< Content-Length: 103
< Elapsed: 00:00.840189
{ 'info': {'subdir': 'noarch'},
'packages': {},
'removed': [],
'repodata_version': 1}
DEBUG conda.core.subdir_data:_pickle_me(284): Saving pickled state for file:///X:/<LOCAL-MIRROR-PATH>/msys2/noarch/repodata.json at C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3\pkgs\cache\e8bdfbda.json
INFO conda._vendor.auxlib.logz:stringify(144): request is 'None' for Response object with url file:///X:/<LOCAL-MIRROR-PATH>/msys2/win-64/repodata.json
DEBUG conda.core.subdir_data:fetch_repodata_remote_request(487):
<<FILE 200 None
< Content-Type: application/json
< Last-Modified: Thu, 30 Jan 2020 00:18:12 GMT
< Content-Length: 227211
< Elapsed: 00:00.843118
{ 'info': {'subdir': 'win-64'},
'packages': { 'm2-autoconf-2.69-3.tar.bz2': { 'build': '3',
'build_number': 3,
'depends': [ 'm2-bash',
INFO conda._vendor.auxlib.logz:stringify(144): request is 'None' for Response object with url file:///X:/<LOCAL-MIRROR-PATH>/main/noarch/repodata.json
DEBUG conda.core.subdir_data:_pickle_me(284): Saving pickled state for file:///X:/<LOCAL-MIRROR-PATH>/msys2/win-64/repodata.json at C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3\pkgs\cache\270cd9d6.json
DEBUG conda.core.subdir_data:fetch_repodata_remote_request(487):
<<FILE 200 None
< Content-Type: application/json
< Last-Modified: Thu, 30 Jan 2020 00:17:55 GMT
< Content-Length: 704400
< Elapsed: 00:00.841911
{ 'info': {'subdir': 'noarch'},
'packages': { 'affine-2.1.0-pyh128a3a6_1.tar.bz2': { 'build': 'pyh128a3a6_1',
'build_number': 1,
'depends': ['py
conda.core.prefix_data.PrefixData.load,0.565156
conda.core.solve.Solver._collect_all_metadata,119.613876
conda.core.prefix_data.PrefixData.load,0.163001
conda.core.solve.Solver._collect_all_metadata,9.134097
conda.core.prefix_data.PrefixData.load,0.161001
conda.core.solve.Solver._collect_all_metadata,84.478895
Environment Information
`conda info`
active environment : None
shell level : 0
user config file : C:\Users\<USERNAME>\.condarc
populated config files : C:\Users\<USERNAME>\.condarc
conda version : 4.8.1
conda-build version : 3.18.11
python version : 3.7.6.final.0
virtual packages :
base environment : C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3 (writable)
channel URLs : file:///X:/<LOCAL-MIRROR-PATH>/main/win-64
file:///X:/<LOCAL-MIRROR-PATH>/main/noarch
file:///X:/<LOCAL-MIRROR-PATH>/msys2/win-64
file:///X:/<LOCAL-MIRROR-PATH>/msys2/noarch
package cache : C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3\pkgs
C:\Users\<USERNAME>\.conda\pkgs
C:\Users\<USERNAME>\AppData\Local\conda\conda\pkgs
envs directories : C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3\envs
C:\Users\<USERNAME>\.conda\envs
C:\Users\<USERNAME>\AppData\Local\conda\conda\envs
platform : win-64
user-agent : conda/4.8.1 requests/2.22.0 CPython/3.7.6 Windows/10 Windows/10.0.17763
administrator : False
netrc file : None
offline mode : False
`conda config --show-sources`
==> C:\Users\<USERNAME>\.condarc <==
auto_update_conda: False
auto_activate_base: False
channel_alias: file:///X:/<LOCAL-MIRROR-PATH>
channel_priority: disabled
channels:
- defaults
default_channels:
- main
- msys2
show_channel_urls: True
use_only_tar_bz2: True
report_errors: False
`conda list --show-channel-urls`
# packages in environment at C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3:
#
# Name Version Build Channel
asn1crypto 1.3.0 py37_0 defaults
backports 1.0 py_2 defaults
backports.functools_lru_cache 1.6.1 py_0 defaults
backports.tempfile 1.0 py_1 defaults
backports.weakref 1.0.post1 py_1 defaults
beautifulsoup4 4.8.2 py37_0 defaults
bzip2 1.0.8 he774522_0 defaults
ca-certificates 2019.11.27 0 defaults
certifi 2019.11.28 py37_0 defaults
cffi 1.13.2 py37h7a1dbc1_0 defaults
chardet 3.0.4 py37_1003 defaults
click 7.0 py_0 defaults
conda 4.8.1 py37_0 defaults
conda-build 3.18.11 py37_0 defaults
conda-package-handling 1.6.0 py37h62dcd97_0 defaults
conda-verify 3.4.2 py_1 defaults
console_shortcut 0.1.1 3 https://repo.anaconda.com/pkgs/main
constructor 3.0.0 py37_0 defaults
cryptography 2.8 py37h7a1dbc1_0 defaults
filelock 3.0.12 py_0 defaults
freetype 2.9.1 ha9979f8_1 defaults
future 0.18.2 py37_0 defaults
glob2 0.7 py_0 defaults
idna 2.8 py37_0 https://repo.anaconda.com/pkgs/main
jinja2 2.10.3 py_0 defaults
jpeg 9b hb83a4c4_2 defaults
libarchive 3.3.3 h0643e63_5 defaults
libiconv 1.15 h1df5818_7 defaults
liblief 0.9.0 ha925a31_2 defaults
libpng 1.6.37 h2a8f88b_0 defaults
libtiff 4.1.0 h56a325e_0 defaults
libxml2 2.9.9 h464c3ec_0 defaults
lz4-c 1.8.1.2 h2fa13f4_0 defaults
lzo 2.10 h6df0209_2 defaults
m2-msys2-runtime 2.5.0.17080.65c939c 3 defaults
m2-patch 2.7.5 2 defaults
markupsafe 1.1.1 py37he774522_0 defaults
menuinst 1.4.16 py37he774522_0 https://repo.anaconda.com/pkgs/main
msys2-conda-epoch 20160418 1 defaults
nsis 3.01 8 defaults
olefile 0.46 py37_0 defaults
openssl 1.1.1d he774522_3 defaults
pillow 7.0.0 py37hcc1f983_0 defaults
pip 20.0.2 py37_0 defaults
pkginfo 1.5.0.1 py37_0 defaults
powershell_shortcut 0.0.1 2 https://repo.anaconda.com/pkgs/main
psutil 5.6.7 py37he774522_0 defaults
py-lief 0.9.0 py37ha925a31_2 defaults
pycosat 0.6.3 py37he774522_0 defaults
pycparser 2.19 py_0 defaults
pyopenssl 19.1.0 py37_0 defaults
pysocks 1.7.1 py37_0 defaults
python 3.7.6 h60c2a47_2 defaults
python-dateutil 2.8.1 py_0 defaults
python-libarchive-c 2.8 py37_13 defaults
pytz 2019.3 py_0 defaults
pywin32 227 py37he774522_1 defaults
pyyaml 5.2 py37he774522_0 defaults
requests 2.22.0 py37_1 defaults
ruamel_yaml 0.15.87 py37he774522_0 defaults
setuptools 45.1.0 py37_0 defaults
six 1.14.0 py37_0 defaults
soupsieve 1.9.5 py37_0 defaults
sqlite 3.30.1 he774522_0 defaults
tk 8.6.8 hfa6e2cd_0 defaults
tqdm 4.42.0 py_0 defaults
urllib3 1.25.8 py37_0 defaults
vc 14.1 h0510ff6_4 https://repo.anaconda.com/pkgs/main
vs2015_runtime 14.16.27012 hf0eaf9b_1 defaults
wheel 0.33.6 py37_0 defaults
win_inet_pton 1.1.0 py37_0 https://repo.anaconda.com/pkgs/main
wincertstore 0.2 py37_0 https://repo.anaconda.com/pkgs/main
xz 5.2.4 h2fa13f4_4 defaults
yaml 0.1.7 hc54c509_2 https://repo.anaconda.com/pkgs/main
zlib 1.2.11 h62dcd97_3 defaults
zstd 1.3.7 h508b16e_0 defaults