Import speedups #7122

mbargull · 2018-04-04T08:55:43Z

conda's startup time is heavily influenced by some costly imports of external modules.
This PR alone cuts the overall import time of conda.base.context by approx. 25 % (using conda 4.5.0+python 3.6.5).

One costly import we cannot avoid is ruamel.yaml/ruamel_yaml. 1/3 of the import time* of that package is spent compiling regular expression (of which usually only few are actually used). We could monkeypatch re.compile to avoid this, but the saner approach is of course to fix this upstream. This is done in https://bitbucket.org/ruamel/yaml/pull-requests/27/improve-package-import-time/diff.

>>> t_base = 0.01278
>>> t_ruamel_unpatched = 0.04364
>>> t_ruamel_patched = 0.03291
>>> (t_ruamel_patched - t_base) / (t_ruamel_unpatched - t_base)
0.6523007128969541

If that ruamel.yaml PR gets accepted, the overall time savings combined with this PR would be approx. 33 %.

 runtime in seconds (percentiles of 1000 runs)  | shell command
      0        25        50        75       100 |
------------------------------------------------|-------------------------------------------------
## base
------------------------------------------------|-------------------------------------------------
0.00169   0.00182   0.00185   0.00188   0.00501 | true
0.01236   0.01270   0.01278   0.01289   0.01729 | python -c 'pass'
------------------------------------------------|-------------------------------------------------
## base conda imports
------------------------------------------------|-------------------------------------------------
0.02038   0.02080   0.02095   0.02118   0.03383 | python -c 'import conda'
0.02046   0.02089   0.02103   0.02122   0.02558 | python -c 'import conda.base'
0.02030   0.02082   0.02096   0.02117   0.02652 | python -c 'import conda.common'
------------------------------------------------|-------------------------------------------------
## external imports
------------------------------------------------|-------------------------------------------------
0.02504   0.02579   0.02599   0.02632   0.03171 | python -c 'import inspect'
0.01254   0.01285   0.01292   0.01303   0.01771 | python -c 'import urllib'
0.04746   0.04873   0.05025   0.05266   0.05886 | python -c 'import urllib.request'
0.02052   0.02090   0.02104   0.02126   0.02633 | python -c 'import conda._vendor
0.02057   0.02105   0.02121   0.02140   0.02667 | python -c 'import conda._vendor.toolz'
0.03012   0.03086   0.03110   0.03162   0.03635 | python -c 'import conda._vendor.toolz.functoolz'
0.04172   0.04272   0.04364   0.04503   0.05020 | python -c 'import ruamel_yaml'
------------------------------------------------|-------------------------------------------------
## conda modules with costly imports
------------------------------------------------|-------------------------------------------------
0.05470   0.05590   0.05764   0.06027   0.06399 | python -c 'import conda.common.path'
0.05579   0.05742   0.05905   0.06177   0.07585 | python -c 'import conda.common.url'
0.05349   0.05526   0.05712   0.05940   0.06720 | python -c 'import conda.common.serialize'
0.08788   0.09484   0.09662   0.09722   0.10350 | python -c 'import conda.common.configuration'
0.09521   0.10202   0.10533   0.10595   0.11980 | python -c 'import conda.base.context'
------------------------------------------------|-------------------------------------------------
## after applying this PR
------------------------------------------------|-------------------------------------------------
0.03354   0.03420   0.03459   0.03518   0.04024 | python -c 'import conda.common.path'
0.03711   0.03804   0.03854   0.03938   0.04418 | python -c 'import conda.common.url'
0.06232   0.06420   0.06715   0.06926   0.07643 | python -c 'import conda.common.configuration'
0.07469   0.07793   0.08171   0.08296   0.08962 | python -c 'import conda.base.context'
------------------------------------------------|-------------------------------------------------
## this PR + https://bitbucket.org/ruamel/yaml/pull-requests/27/improve-package-import-time
------------------------------------------------|-------------------------------------------------
0.03147   0.03242   0.03291   0.03406   0.04619 | python -c 'import ruamel_yaml'
0.04366   0.04494   0.04639   0.04814   0.05386 | python -c 'import conda.common.serialize'
0.05225   0.05371   0.05651   0.05808   0.08065 | python -c 'import conda.common.configuration'
0.06442   0.06807   0.07103   0.07165   0.08386 | python -c 'import conda.base.context'
------------------------------------------------|-------------------------------------------------

kalefranz · 2018-04-04T17:13:27Z

This is awesome. Thanks for poking around here @mbargull.

kalefranz · 2018-04-04T17:17:02Z

Let's see how the upstream patch goes, and then decide what to do about ruamel.yaml after that...

kalefranz · 2018-04-04T17:19:07Z

Note this is targeted at master, which is conda 4.6. I don't think that's an issue. I'd like to get 4.6 release relatively soon. What would mark a 4.6 release is the inclusion of #6518.

kalefranz · 2018-04-04T17:20:27Z

Also, just throwing this out there... Seems like the world is moving toward toml. Wonder what it would look like for us to start to handle both yaml and toml, eventually moving away from yaml? Just a thought...

kalefranz · 2018-04-04T17:30:02Z

Is it worth mucking around with

conda/conda/common/serialize.py

Line 28 in 31130cd

yaml = get_yaml()

to try to lazy-load that? Basically lazy-load any ruamel.yaml import. It would help users with no yaml configuration at all. I'm not sure if it's worth it or not...

kalefranz · 2018-04-04T17:31:37Z

BTW, how did you generate that report? It's very nice. Mind sharing the code?

kalefranz

I'm fine with this PR as-is. Maybe in the future we could look at lazy-loading ruamel.yaml as a whole, but not all users will benefit from that, so I'm fine not adding in that complexity also.

mbargull · 2018-04-04T19:01:26Z

Basically lazy-load any ruamel.yaml import. It would help users with no yaml configuration at all. I'm not sure if it's worth it or not...

Yeah, I agree with "not sure if it's worth it or not" and "I'm fine not adding in that complexity also." 😉

BTW, how did you generate that report? It's very nice. Mind sharing the code?

Nothing fancy really, just a little home-brewed script:

#!/usr/bin/env python

from __future__ import print_function

from itertools import chain
from os import system
from timeit import repeat

repeats = 1000
row_sep = "-"
col_sep = "|"
percentiles = (0, .25, .5, .75, 1)

shell_commands = (
    "true",
    "python -c 'pass'",
    *(
        "python -c 'import {}'".format(module)
        for module in (
            "conda",
            "conda.base",
            "conda.common",
            "inspect",
            "urllib",
            "urllib.request",
            "conda._vendor",
            "conda._vendor.toolz",
            "conda._vendor.toolz.functoolz",
        )
    ),
    *(chain.from_iterable(
        (
            """python -c 'import re'""".format(module),
            """python -c 'import {}'""".format(module),
            """python -c 'import re; dummy=re.compile(""); re.compile = lambda *args: dummy; import {}'""".format(module),
        )
        for module in ("ruamel_yaml",)
    )),
)

def to_percentile_string(values, percentiles=percentiles):
    values = sorted(values)
    percentile_values = (values[int(p * (len(values)-1))] for p in percentiles)
    return " ".join("{:9.5f}".format(v) for v in percentile_values)

for shell_command in shell_commands:
    times = repeat(lambda: system(shell_command), number=1, repeat=repeats)
    print(to_percentile_string(times), col_sep, shell_command)
print(row_sep * len(to_percentile_string((0,))), col_sep, row_sep * max(map(len, shell_commands)), sep=row_sep)

mbargull · 2018-06-17T09:54:07Z

Let's see how the upstream patch goes, and then decide what to do about ruamel.yaml after that...

https://pypi.org/project/ruamel.yaml/0.15.39/#changelog

0.15.39 (2018-06-17):

merge PR27 improving package startup time (and loading when regexp not actually used)

🎉

mbargull · 2018-06-17T14:57:40Z

Since y'all liked the quartiles of measured run times so much, here are some new ones:

TLDR; conda activate will take only 59 % of the previous run time when everything is updated to their latest (pre-) release versions.

python 3.6.5
conda 4.5.4
ruamel.yaml 0.15.38
  0.20428   0.20925   0.21042   0.21285   0.22077 | python -m conda info
  0.10103   0.10310   0.10370   0.10443   0.10866 | python -m conda --version
  0.11157   0.11465   0.11562   0.11659   0.12716 | python -m conda shell.posix activate

python 3.7.0b5
conda 4.6.0a2
ruamel.yaml 0.15.39
  0.16274   0.16682   0.16821   0.16980   0.18873 | python -m conda info
  0.07340   0.07514   0.07583   0.07690   0.08124 | python -m conda --version
  0.06575   0.06800   0.06866   0.06926   0.07579 | python -m conda shell.posix activate

full output:

docker run --rm -it python:3.6-slim bash -c "$(cat <<'$$$$'
(
cat >./time-conda-from-shell <<'EOF'
#!/usr/bin/env python

from __future__ import print_function

from collections import OrderedDict
from itertools import chain
from os import system
from timeit import repeat


repeats = 100
row_sep = '-'
col_sep = '|'
percentiles = (0, .25, .5, .75, 1)


def import_shell_command(module):
    return "python -c 'import {}'".format(module)


runs = OrderedDict((
    ('base', ['true', "python -c 'pass'"]),
    ('base conda imports', list(map(import_shell_command, (
        'conda',
        'conda.base',
        'conda.common',
    )))),
    ('external imports', list(map(import_shell_command, (
        'inspect',
        'urllib',
        'urllib.request',
        'conda._vendor',
        'conda._vendor.toolz',
        'conda._vendor.toolz.functoolz',
        'ruamel.yaml',
    )))),
    ('conda modules with costly imports', list(map(import_shell_command, (
        'conda.common.path',
        'conda.common.url',
        'conda.common.serialize',
        'conda.common.configuration',
        'conda.base.context',
    )))),
    ('conda commands', [
        'python -m conda info',
        'python -m conda --version',
        'python -m conda shell.posix activate',
    ]),
))

def to_percentile_string(values, percentiles=percentiles, fmt='{:9.5f}'):
    values = sorted(values)
    # NOTE: Don't really care if we get a one-off-error in this context...
    percentile_values = (values[int(p * (len(values)-1))] for p in percentiles)
    return ' '.join(fmt.format(v) for v in percentile_values)


def print_sep_row(
    row_sep=row_sep, col_sep=col_sep,
    _row_len=max(max(map(len, cmds)) for cmds in runs.values())
):
    print(
        row_sep * len(to_percentile_string((0,))),
        col_sep,
        row_sep * _row_len,
        sep=row_sep,
    )


def run(category, shell_commands, print_category=True, print_header=False):
    if print_header:
        print(
            '{0:^{1}}'.format(
                f'  runtime in seconds (percentiles of {repeats} runs)',
                len(to_percentile_string((0,))),
            ),
            col_sep, 'shell command'
        )
        print(to_percentile_string(range(101), fmt='{:>9}'), col_sep)
        print_sep_row()
    if print_category:
        print('##', category)
        print_sep_row()

    for shell_command in shell_commands:
        silent_shell_command = f'{shell_command} >/dev/null 2>&1'
        times = repeat(
            lambda: system(silent_shell_command), number=1, repeat=repeats,
        )
        print(to_percentile_string(times), col_sep, shell_command)
    if print_category:
        print_sep_row()


for i, (category, shell_commands) in enumerate(runs.items()):
    run(category, shell_commands, print_header=(i == 0))
EOF
chmod +x ./time-conda-from-shell
apt-get update -qq && apt-get install -qqy --no-install-recommends git g++ >/dev/null 2>&1
pip install -q ruamel.yaml==0.15.38
pip install -q git+https://github.com/conda/conda@4.5.4
python -c 'import sys,conda,ruamel.yaml; print(f"python {sys.version.split()[0]}\nconda {conda.__version__}\nruamel.yaml {ruamel.yaml.__version__}")' && ./time-conda-from-shell
pip install -q ruamel.yaml==0.15.39
python -c 'import sys,conda,ruamel.yaml; print(f"python {sys.version.split()[0]}\nconda {conda.__version__}\nruamel.yaml {ruamel.yaml.__version__}")' && ./time-conda-from-shell
pip install -q ruamel.yaml==0.15.38
pip install -q git+https://github.com/conda/conda@4.6.0a2
python -c 'import sys,conda,ruamel.yaml; print(f"python {sys.version.split()[0]}\nconda {conda.__version__}\nruamel.yaml {ruamel.yaml.__version__}")' && ./time-conda-from-shell
pip install -q ruamel.yaml==0.15.39
python -c 'import sys,conda,ruamel.yaml; print(f"python {sys.version.split()[0]}\nconda {conda.__version__}\nruamel.yaml {ruamel.yaml.__version__}")' && ./time-conda-from-shell
)
$$$$
)"

python 3.6.5
conda 4.5.4
ruamel.yaml 0.15.38
   runtime in seconds (percentiles of 100 runs)   | shell command
        0        25        50        75       100 |
--------------------------------------------------|-------------------------------------------------
## base
--------------------------------------------------|-------------------------------------------------
  0.00038   0.00048   0.00049   0.00053   0.00075 | true
  0.01627   0.01662   0.01676   0.01696   0.01997 | python -c 'pass'
--------------------------------------------------|-------------------------------------------------
## base conda imports
--------------------------------------------------|-------------------------------------------------
  0.02125   0.02179   0.02196   0.02214   0.02412 | python -c 'import conda'
  0.02168   0.02199   0.02218   0.02251   0.02703 | python -c 'import conda.base'
  0.02140   0.02185   0.02198   0.02231   0.02435 | python -c 'import conda.common'
--------------------------------------------------|-------------------------------------------------
## external imports
--------------------------------------------------|-------------------------------------------------
  0.02701   0.02763   0.02789   0.02842   0.03242 | python -c 'import inspect'
  0.01658   0.01693   0.01734   0.01795   0.03579 | python -c 'import urllib'
  0.05299   0.05383   0.05414   0.05508   0.05905 | python -c 'import urllib.request'
  0.02161   0.02187   0.02214   0.02242   0.02721 | python -c 'import conda._vendor'
  0.02153   0.02210   0.02225   0.02255   0.02718 | python -c 'import conda._vendor.toolz'
  0.03278   0.03327   0.03346   0.03390   0.03613 | python -c 'import conda._vendor.toolz.functoolz'
  0.04449   0.04537   0.04589   0.04652   0.05361 | python -c 'import ruamel.yaml'
--------------------------------------------------|-------------------------------------------------
## conda modules with costly imports
--------------------------------------------------|-------------------------------------------------
  0.06151   0.06221   0.06294   0.06366   0.07883 | python -c 'import conda.common.path'
  0.06261   0.06385   0.06439   0.06508   0.07171 | python -c 'import conda.common.url'
  0.05864   0.06037   0.06077   0.06148   0.07303 | python -c 'import conda.common.serialize'
  0.09866   0.10113   0.10217   0.10329   0.11008 | python -c 'import conda.common.configuration'
  0.10843   0.11122   0.11222   0.11354   0.12386 | python -c 'import conda.base.context'
--------------------------------------------------|-------------------------------------------------
## conda commands
--------------------------------------------------|-------------------------------------------------
  0.20428   0.20925   0.21042   0.21285   0.22077 | python -m conda info
  0.10103   0.10310   0.10370   0.10443   0.10866 | python -m conda --version
  0.11157   0.11465   0.11562   0.11659   0.12716 | python -m conda shell.posix activate
--------------------------------------------------|-------------------------------------------------
python 3.6.5
conda 4.5.4
ruamel.yaml 0.15.39
   runtime in seconds (percentiles of 100 runs)   | shell command
        0        25        50        75       100 |
--------------------------------------------------|-------------------------------------------------
## base
--------------------------------------------------|-------------------------------------------------
  0.00037   0.00048   0.00049   0.00055   0.00095 | true
  0.01641   0.01666   0.01677   0.01710   0.02183 | python -c 'pass'
--------------------------------------------------|-------------------------------------------------
## base conda imports
--------------------------------------------------|-------------------------------------------------
  0.02216   0.02257   0.02289   0.02322   0.02533 | python -c 'import conda'
  0.02217   0.02266   0.02302   0.02334   0.02516 | python -c 'import conda.base'
  0.02210   0.02264   0.02294   0.02325   0.02630 | python -c 'import conda.common'
--------------------------------------------------|-------------------------------------------------
## external imports
--------------------------------------------------|-------------------------------------------------
  0.02785   0.02856   0.02895   0.02943   0.03456 | python -c 'import inspect'
  0.01673   0.01715   0.01736   0.01770   0.01983 | python -c 'import urllib'
  0.05421   0.05512   0.05570   0.05646   0.05972 | python -c 'import urllib.request'
  0.02204   0.02242   0.02260   0.02287   0.02663 | python -c 'import conda._vendor'
  0.02222   0.02275   0.02305   0.02358   0.02860 | python -c 'import conda._vendor.toolz'
  0.03359   0.03418   0.03451   0.03510   0.03745 | python -c 'import conda._vendor.toolz.functoolz'
  0.03319   0.03373   0.03399   0.03435   0.03798 | python -c 'import ruamel.yaml'
--------------------------------------------------|-------------------------------------------------
## conda modules with costly imports
--------------------------------------------------|-------------------------------------------------
  0.06240   0.06353   0.06399   0.06487   0.07036 | python -c 'import conda.common.path'
  0.06415   0.06548   0.06597   0.06720   0.07684 | python -c 'import conda.common.url'
  0.04813   0.04868   0.04905   0.04957   0.05264 | python -c 'import conda.common.serialize'
  0.08830   0.08939   0.09014   0.09110   0.10340 | python -c 'import conda.common.configuration'
  0.09534   0.09847   0.09931   0.10015   0.11062 | python -c 'import conda.base.context'
--------------------------------------------------|-------------------------------------------------
## conda commands
--------------------------------------------------|-------------------------------------------------
  0.19257   0.19657   0.19803   0.20137   0.21605 | python -m conda info
  0.10220   0.10309   0.10442   0.10538   0.10971 | python -m conda --version
  0.10006   0.10224   0.10317   0.10422   0.10935 | python -m conda shell.posix activate
--------------------------------------------------|-------------------------------------------------
python 3.6.5
conda 4.6.0a2
ruamel.yaml 0.15.38
   runtime in seconds (percentiles of 100 runs)   | shell command
        0        25        50        75       100 |
--------------------------------------------------|-------------------------------------------------
## base
--------------------------------------------------|-------------------------------------------------
  0.00037   0.00047   0.00049   0.00052   0.00066 | true
  0.01630   0.01653   0.01671   0.01700   0.02026 | python -c 'pass'
--------------------------------------------------|-------------------------------------------------
## base conda imports
--------------------------------------------------|-------------------------------------------------
  0.02127   0.02176   0.02190   0.02226   0.02635 | python -c 'import conda'
  0.02163   0.02190   0.02210   0.02236   0.02667 | python -c 'import conda.base'
  0.02138   0.02180   0.02197   0.02226   0.02592 | python -c 'import conda.common'
--------------------------------------------------|-------------------------------------------------
## external imports
--------------------------------------------------|-------------------------------------------------
  0.02692   0.02746   0.02772   0.02823   0.03201 | python -c 'import inspect'
  0.01638   0.01685   0.01709   0.01727   0.01921 | python -c 'import urllib'
  0.05320   0.05422   0.05483   0.05561   0.06875 | python -c 'import urllib.request'
  0.02168   0.02241   0.02266   0.02323   0.02719 | python -c 'import conda._vendor'
  0.02194   0.02257   0.02281   0.02318   0.02594 | python -c 'import conda._vendor.toolz'
  0.03335   0.03516   0.03632   0.03928   0.04326 | python -c 'import conda._vendor.toolz.functoolz'
  0.04597   0.04658   0.04731   0.04804   0.05276 | python -c 'import ruamel.yaml'
--------------------------------------------------|-------------------------------------------------
## conda modules with costly imports
--------------------------------------------------|-------------------------------------------------
  0.03823   0.03996   0.04129   0.04368   0.04864 | python -c 'import conda.common.path'
  0.04255   0.04317   0.04370   0.04428   0.04827 | python -c 'import conda.common.url'
  0.06085   0.06350   0.06707   0.07075   0.08218 | python -c 'import conda.common.serialize'
  0.06988   0.07270   0.07355   0.08002   0.09328 | python -c 'import conda.common.configuration'
  0.08479   0.08790   0.09060   0.10050   0.10604 | python -c 'import conda.base.context'
--------------------------------------------------|-------------------------------------------------
## conda commands
--------------------------------------------------|-------------------------------------------------
  0.20472   0.20828   0.20952   0.21119   0.23196 | python -m conda info
  0.08313   0.08563   0.08652   0.08737   0.10515 | python -m conda --version
  0.08794   0.09059   0.09173   0.09345   0.10215 | python -m conda shell.posix activate
--------------------------------------------------|-------------------------------------------------
python 3.6.5
conda 4.6.0a2
ruamel.yaml 0.15.39
   runtime in seconds (percentiles of 100 runs)   | shell command
        0        25        50        75       100 |
--------------------------------------------------|-------------------------------------------------
## base
--------------------------------------------------|-------------------------------------------------
  0.00044   0.00046   0.00054   0.00057   0.00096 | true
  0.01676   0.01718   0.01757   0.01823   0.03176 | python -c 'pass'
--------------------------------------------------|-------------------------------------------------
## base conda imports
--------------------------------------------------|-------------------------------------------------
  0.02177   0.02234   0.02252   0.02279   0.02771 | python -c 'import conda'
  0.02161   0.02222   0.02246   0.02275   0.02680 | python -c 'import conda.base'
  0.02188   0.02234   0.02256   0.02311   0.02803 | python -c 'import conda.common'
--------------------------------------------------|-------------------------------------------------
## external imports
--------------------------------------------------|-------------------------------------------------
  0.02732   0.02829   0.02855   0.02895   0.03290 | python -c 'import inspect'
  0.01666   0.01723   0.01741   0.01760   0.02204 | python -c 'import urllib'
  0.05431   0.05520   0.05578   0.05648   0.06135 | python -c 'import urllib.request'
  0.02158   0.02236   0.02258   0.02283   0.02753 | python -c 'import conda._vendor'
  0.02199   0.02260   0.02276   0.02304   0.02554 | python -c 'import conda._vendor.toolz'
  0.03306   0.03405   0.03447   0.03476   0.04017 | python -c 'import conda._vendor.toolz.functoolz'
  0.03347   0.03402   0.03431   0.03467   0.03958 | python -c 'import ruamel.yaml'
--------------------------------------------------|-------------------------------------------------
## conda modules with costly imports
--------------------------------------------------|-------------------------------------------------
  0.03772   0.03875   0.03901   0.03938   0.04451 | python -c 'import conda.common.path'
  0.04219   0.04304   0.04332   0.04390   0.04936 | python -c 'import conda.common.url'
  0.04698   0.04853   0.04904   0.04987   0.05580 | python -c 'import conda.common.serialize'
  0.05922   0.05994   0.06041   0.06123   0.07111 | python -c 'import conda.common.configuration'
  0.07273   0.07411   0.07488   0.07586   0.08044 | python -c 'import conda.base.context'
--------------------------------------------------|-------------------------------------------------
## conda commands
--------------------------------------------------|-------------------------------------------------
  0.19172   0.19551   0.19719   0.19875   0.21032 | python -m conda info
  0.08395   0.08580   0.08646   0.08741   0.10541 | python -m conda --version
  0.07569   0.07781   0.07877   0.08193   0.10352 | python -m conda shell.posix activate
--------------------------------------------------|-------------------------------------------------

docker run --rm -it python:3.7-rc-slim bash -c "$(cat <<'$$$$'
(
cat >./time-conda-from-shell <<'EOF'
#!/usr/bin/env python

from __future__ import print_function

from collections import OrderedDict
from itertools import chain
from os import system
from timeit import repeat


repeats = 100
row_sep = '-'
col_sep = '|'
percentiles = (0, .25, .5, .75, 1)


def import_shell_command(module):
    return "python -c 'import {}'".format(module)


runs = OrderedDict((
    ('base', ['true', "python -c 'pass'"]),
    ('base conda imports', list(map(import_shell_command, (
        'conda',
        'conda.base',
        'conda.common',
    )))),
    ('external imports', list(map(import_shell_command, (
        'inspect',
        'urllib',
        'urllib.request',
        'conda._vendor',
        'conda._vendor.toolz',
        'conda._vendor.toolz.functoolz',
        'ruamel.yaml',
    )))),
    ('conda modules with costly imports', list(map(import_shell_command, (
        'conda.common.path',
        'conda.common.url',
        'conda.common.serialize',
        'conda.common.configuration',
        'conda.base.context',
    )))),
    ('conda commands', [
        'python -m conda info',
        'python -m conda --version',
        'python -m conda shell.posix activate',
    ]),
))

def to_percentile_string(values, percentiles=percentiles, fmt='{:9.5f}'):
    values = sorted(values)
    # NOTE: Don't really care if we get a one-off-error in this context...
    percentile_values = (values[int(p * (len(values)-1))] for p in percentiles)
    return ' '.join(fmt.format(v) for v in percentile_values)


def print_sep_row(
    row_sep=row_sep, col_sep=col_sep,
    _row_len=max(max(map(len, cmds)) for cmds in runs.values())
):
    print(
        row_sep * len(to_percentile_string((0,))),
        col_sep,
        row_sep * _row_len,
        sep=row_sep,
    )


def run(category, shell_commands, print_category=True, print_header=False):
    if print_header:
        print(
            '{0:^{1}}'.format(
                f'  runtime in seconds (percentiles of {repeats} runs)',
                len(to_percentile_string((0,))),
            ),
            col_sep, 'shell command'
        )
        print(to_percentile_string(range(101), fmt='{:>9}'), col_sep)
        print_sep_row()
    if print_category:
        print('##', category)
        print_sep_row()

    for shell_command in shell_commands:
        silent_shell_command = f'{shell_command} >/dev/null 2>&1'
        times = repeat(
            lambda: system(silent_shell_command), number=1, repeat=repeats,
        )
        print(to_percentile_string(times), col_sep, shell_command)
    if print_category:
        print_sep_row()


for i, (category, shell_commands) in enumerate(runs.items()):
    run(category, shell_commands, print_header=(i == 0))
EOF
chmod +x ./time-conda-from-shell
apt-get update -qq && apt-get install -qqy --no-install-recommends git g++ >/dev/null 2>&1
pip install -q ruamel.yaml==0.15.38
pip install -q git+https://github.com/conda/conda@4.5.4
sed -i 's/\(f\.__name__ for f in reversed((self\.first,) + self\.funcs)\),/\1/' /usr/local/lib/python3.7/site-packages/conda/_vendor/toolz/functoolz.py 
python -c 'import sys,conda,ruamel.yaml; print(f"python {sys.version.split()[0]}\nconda {conda.__version__}\nruamel.yaml {ruamel.yaml.__version__}")' && ./time-conda-from-shell
pip install -q ruamel.yaml==0.15.39
python -c 'import sys,conda,ruamel.yaml; print(f"python {sys.version.split()[0]}\nconda {conda.__version__}\nruamel.yaml {ruamel.yaml.__version__}")' && ./time-conda-from-shell
pip install -q ruamel.yaml==0.15.38
pip install -q git+https://github.com/conda/conda@4.6.0a2
sed -i 's/\(f\.__name__ for f in reversed((self\.first,) + self\.funcs)\),/\1/' /usr/local/lib/python3.7/site-packages/conda/_vendor/toolz/functoolz.py 
python -c 'import sys,conda,ruamel.yaml; print(f"python {sys.version.split()[0]}\nconda {conda.__version__}\nruamel.yaml {ruamel.yaml.__version__}")' && ./time-conda-from-shell
pip install -q ruamel.yaml==0.15.39
python -c 'import sys,conda,ruamel.yaml; print(f"python {sys.version.split()[0]}\nconda {conda.__version__}\nruamel.yaml {ruamel.yaml.__version__}")' && ./time-conda-from-shell
)
$$$$
)"

python 3.7.0b5
conda 4.5.4
ruamel.yaml 0.15.38
   runtime in seconds (percentiles of 100 runs)   | shell command
        0        25        50        75       100 |
--------------------------------------------------|-------------------------------------------------
## base
--------------------------------------------------|-------------------------------------------------
  0.00037   0.00047   0.00049   0.00055   0.00133 | true
  0.01493   0.01532   0.01543   0.01564   0.01944 | python -c 'pass'
--------------------------------------------------|-------------------------------------------------
## base conda imports
--------------------------------------------------|-------------------------------------------------
  0.01998   0.02031   0.02071   0.02106   0.02964 | python -c 'import conda'
  0.02008   0.02051   0.02085   0.02130   0.02500 | python -c 'import conda.base'
  0.01991   0.02038   0.02052   0.02086   0.02487 | python -c 'import conda.common'
--------------------------------------------------|-------------------------------------------------
## external imports
--------------------------------------------------|-------------------------------------------------
  0.02101   0.02137   0.02158   0.02181   0.02894 | python -c 'import inspect'
  0.01527   0.01555   0.01568   0.01591   0.01989 | python -c 'import urllib'
  0.04521   0.04582   0.04599   0.04625   0.05458 | python -c 'import urllib.request'
  0.02008   0.02050   0.02070   0.02098   0.02596 | python -c 'import conda._vendor'
  0.02023   0.02058   0.02072   0.02101   0.02780 | python -c 'import conda._vendor.toolz'
  0.02619   0.02668   0.02691   0.02720   0.03832 | python -c 'import conda._vendor.toolz.functoolz'
  0.04070   0.04120   0.04157   0.04207   0.04760 | python -c 'import ruamel.yaml'
--------------------------------------------------|-------------------------------------------------
## conda modules with costly imports
--------------------------------------------------|-------------------------------------------------
  0.05269   0.05337   0.05376   0.05471   0.05906 | python -c 'import conda.common.path'
  0.05406   0.05516   0.05595   0.05723   0.06250 | python -c 'import conda.common.url'
  0.05330   0.05425   0.05477   0.05549   0.06050 | python -c 'import conda.common.serialize'
  0.08410   0.08510   0.08597   0.08705   0.09549 | python -c 'import conda.common.configuration'
  0.09026   0.09251   0.09316   0.09414   0.09993 | python -c 'import conda.base.context'
--------------------------------------------------|-------------------------------------------------
## conda commands
--------------------------------------------------|-------------------------------------------------
  0.17409   0.17788   0.17932   0.18099   0.19470 | python -m conda info
  0.08737   0.08938   0.09004   0.09077   0.09975 | python -m conda --version
  0.09308   0.09504   0.09581   0.09701   0.10789 | python -m conda shell.posix activate
--------------------------------------------------|-------------------------------------------------
python 3.7.0b5
conda 4.5.4
ruamel.yaml 0.15.39
   runtime in seconds (percentiles of 100 runs)   | shell command
        0        25        50        75       100 |
--------------------------------------------------|-------------------------------------------------
## base
--------------------------------------------------|-------------------------------------------------
  0.00038   0.00047   0.00051   0.00061   0.00092 | true
  0.01516   0.01560   0.01594   0.01634   0.02003 | python -c 'pass'
--------------------------------------------------|-------------------------------------------------
## base conda imports
--------------------------------------------------|-------------------------------------------------
  0.02042   0.02096   0.02130   0.02156   0.02509 | python -c 'import conda'
  0.02055   0.02120   0.02153   0.02186   0.02619 | python -c 'import conda.base'
  0.02003   0.02049   0.02083   0.02109   0.02431 | python -c 'import conda.common'
--------------------------------------------------|-------------------------------------------------
## external imports
--------------------------------------------------|-------------------------------------------------
  0.02095   0.02161   0.02189   0.02223   0.02567 | python -c 'import inspect'
  0.01522   0.01556   0.01585   0.01611   0.01991 | python -c 'import urllib'
  0.04638   0.04691   0.04724   0.04795   0.05958 | python -c 'import urllib.request'
  0.02055   0.02140   0.02329   0.02422   0.03295 | python -c 'import conda._vendor'
  0.02017   0.02087   0.02111   0.02153   0.02432 | python -c 'import conda._vendor.toolz'
  0.02663   0.02716   0.02739   0.02774   0.03467 | python -c 'import conda._vendor.toolz.functoolz'
  0.03060   0.03152   0.03175   0.03218   0.03657 | python -c 'import ruamel.yaml'
--------------------------------------------------|-------------------------------------------------
## conda modules with costly imports
--------------------------------------------------|-------------------------------------------------
  0.05331   0.05448   0.05489   0.05529   0.06163 | python -c 'import conda.common.path'
  0.05458   0.05573   0.05632   0.05687   0.06201 | python -c 'import conda.common.url'
  0.04239   0.04351   0.04389   0.04446   0.05425 | python -c 'import conda.common.serialize'
  0.07254   0.07447   0.07548   0.08081   0.09242 | python -c 'import conda.common.configuration'
  0.07984   0.08235   0.08299   0.08386   0.09390 | python -c 'import conda.base.context'
--------------------------------------------------|-------------------------------------------------
## conda commands
--------------------------------------------------|-------------------------------------------------
  0.16495   0.16868   0.17091   0.17538   0.19864 | python -m conda info
  0.08901   0.09270   0.09711   0.10283   0.11541 | python -m conda --version
  0.08384   0.08528   0.08676   0.09113   0.10127 | python -m conda shell.posix activate
--------------------------------------------------|-------------------------------------------------
python 3.7.0b5
conda 4.6.0a2
ruamel.yaml 0.15.38
   runtime in seconds (percentiles of 100 runs)   | shell command
        0        25        50        75       100 |
--------------------------------------------------|-------------------------------------------------
## base
--------------------------------------------------|-------------------------------------------------
  0.00037   0.00044   0.00046   0.00051   0.00070 | true
  0.01512   0.01533   0.01542   0.01565   0.01943 | python -c 'pass'
--------------------------------------------------|-------------------------------------------------
## base conda imports
--------------------------------------------------|-------------------------------------------------
  0.01975   0.02020   0.02039   0.02072   0.02545 | python -c 'import conda'
  0.02016   0.02043   0.02060   0.02107   0.02444 | python -c 'import conda.base'
  0.02001   0.02079   0.02104   0.02166   0.02629 | python -c 'import conda.common'
--------------------------------------------------|-------------------------------------------------
## external imports
--------------------------------------------------|-------------------------------------------------
  0.02149   0.02219   0.02250   0.02291   0.02754 | python -c 'import inspect'
  0.01561   0.01614   0.01641   0.01664   0.01928 | python -c 'import urllib'
  0.04626   0.04794   0.04836   0.04872   0.05921 | python -c 'import urllib.request'
  0.02043   0.02146   0.02180   0.02225   0.03141 | python -c 'import conda._vendor'
  0.02056   0.02113   0.02152   0.02196   0.02868 | python -c 'import conda._vendor.toolz'
  0.02666   0.02712   0.02737   0.02767   0.03633 | python -c 'import conda._vendor.toolz.functoolz'
  0.04145   0.04227   0.04288   0.04373   0.04802 | python -c 'import ruamel.yaml'
--------------------------------------------------|-------------------------------------------------
## conda modules with costly imports
--------------------------------------------------|-------------------------------------------------
  0.03424   0.03523   0.03566   0.03601   0.04227 | python -c 'import conda.common.path'
  0.03847   0.03908   0.03939   0.03983   0.04556 | python -c 'import conda.common.url'
  0.05357   0.05446   0.05505   0.05580   0.06186 | python -c 'import conda.common.serialize'
  0.06308   0.06404   0.06483   0.06665   0.08020 | python -c 'import conda.common.configuration'
  0.07562   0.07634   0.07708   0.07836   0.09222 | python -c 'import conda.base.context'
--------------------------------------------------|-------------------------------------------------
## conda commands
--------------------------------------------------|-------------------------------------------------
  0.17339   0.17751   0.17873   0.18057   0.19809 | python -m conda info
  0.07269   0.07541   0.07611   0.07719   0.08049 | python -m conda --version
  0.07757   0.07878   0.07962   0.08067   0.08689 | python -m conda shell.posix activate
--------------------------------------------------|-------------------------------------------------
python 3.7.0b5
conda 4.6.0a2
ruamel.yaml 0.15.39
   runtime in seconds (percentiles of 100 runs)   | shell command
        0        25        50        75       100 |
--------------------------------------------------|-------------------------------------------------
## base
--------------------------------------------------|-------------------------------------------------
  0.00037   0.00047   0.00048   0.00052   0.00075 | true
  0.01496   0.01549   0.01567   0.01589   0.01867 | python -c 'pass'
--------------------------------------------------|-------------------------------------------------
## base conda imports
--------------------------------------------------|-------------------------------------------------
  0.01994   0.02072   0.02089   0.02113   0.02587 | python -c 'import conda'
  0.02033   0.02084   0.02099   0.02124   0.02476 | python -c 'import conda.base'
  0.02035   0.02119   0.02331   0.02409   0.02526 | python -c 'import conda.common'
--------------------------------------------------|-------------------------------------------------
## external imports
--------------------------------------------------|-------------------------------------------------
  0.02112   0.02198   0.02235   0.02302   0.03099 | python -c 'import inspect'
  0.01532   0.01583   0.01596   0.01622   0.02034 | python -c 'import urllib'
  0.04624   0.04734   0.04907   0.05248   0.05621 | python -c 'import urllib.request'
  0.02088   0.02254   0.02367   0.02423   0.02617 | python -c 'import conda._vendor'
  0.02021   0.02106   0.02126   0.02181   0.02687 | python -c 'import conda._vendor.toolz'
  0.02676   0.02726   0.02756   0.02811   0.03306 | python -c 'import conda._vendor.toolz.functoolz'
  0.03115   0.03251   0.03457   0.03650   0.04401 | python -c 'import ruamel.yaml'
--------------------------------------------------|-------------------------------------------------
## conda modules with costly imports
--------------------------------------------------|-------------------------------------------------
  0.03621   0.03945   0.04070   0.04132   0.04546 | python -c 'import conda.common.path'
  0.03799   0.03914   0.03958   0.03993   0.04385 | python -c 'import conda.common.url'
  0.04193   0.04353   0.04396   0.04456   0.04828 | python -c 'import conda.common.serialize'
  0.05219   0.05317   0.05362   0.05392   0.06227 | python -c 'import conda.common.configuration'
  0.06369   0.06537   0.06601   0.06652   0.07074 | python -c 'import conda.base.context'
--------------------------------------------------|-------------------------------------------------
## conda commands
--------------------------------------------------|-------------------------------------------------
  0.16274   0.16682   0.16821   0.16980   0.18873 | python -m conda info
  0.07340   0.07514   0.07583   0.07690   0.08124 | python -m conda --version
  0.06575   0.06800   0.06866   0.06926   0.07579 | python -m conda shell.posix activate
--------------------------------------------------|-------------------------------------------------

msarahan · 2018-06-17T15:30:41Z

Really nice work, @mbargull. Thanks.

github-actions · 2021-09-02T06:03:00Z

Hi there, thank you for your contribution to Conda!

This pull request has been automatically locked since it has not had recent activity after it was closed.

Please open a new issue or pull request if needed.

mbargull added 6 commits March 12, 2018 20:22

common.path: remove unused import

cb1c8cc

common.url: avoid urllib.request import

71d76a5

models.channel: remove unused import

d875d56

common.configuration: avoid slow toolz.functoolz import

f9e0240

common.serialize: make re.compile lazy for ruamel_yaml import

7356e7c

common.serialize: remove test code

e002c46

mbargull requested a review from a team as a code owner April 4, 2018 08:55

mbargull added source::contributor created by a frequent contributor Ready for Review labels Apr 4, 2018

kalefranz approved these changes Apr 4, 2018

View reviewed changes

kalefranz merged commit dc204a6 into conda:master Apr 4, 2018

kalefranz added this to the 4.6.0 milestone Apr 10, 2018

github-actions bot added the locked [bot] locked due to inactivity label Sep 2, 2021

github-actions bot locked as resolved and limited conversation to collaborators Sep 2, 2021

Import speedups #7122

Import speedups #7122

Uh oh!

Conversation

mbargull commented Apr 4, 2018

Uh oh!

kalefranz commented Apr 4, 2018

Uh oh!

kalefranz commented Apr 4, 2018

Uh oh!

kalefranz commented Apr 4, 2018

Uh oh!

kalefranz commented Apr 4, 2018

Uh oh!

kalefranz commented Apr 4, 2018

Uh oh!

kalefranz commented Apr 4, 2018

Uh oh!

kalefranz left a comment

Choose a reason for hiding this comment

Uh oh!

mbargull commented Apr 4, 2018

Uh oh!

mbargull commented Jun 17, 2018

Uh oh!

mbargull commented Jun 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

msarahan commented Jun 17, 2018

Uh oh!

github-actions bot commented Sep 2, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mbargull commented Jun 17, 2018 •

edited

Loading