@kalefranz We discussed this yesterday, would appreciate your feedback on these changes.

Seems reasonable to me. Are the existing tests sufficient? Is it worthwhile to put a time limit test in that conservatively passes with the new code, but which would fail with the old code?

Seems reasonable to me. Are the existing tests sufficient? Is it worthwhile to put a time limit test in that conservatively passes with the new code, but which would fail with the old code?

A few tests are needed to replace the use of CompilePycAction. done

I'd also need to test the compile_multiple_pyc function on Windows to see if there is a size limit on stdin in that platform.

stdin is a pipe on Windows so works much the same, no limit.

The integration tests for pyc compiling on Windows are failing because the .py file has not been linked into the environment when the compilation step is called:

DEBUG conda.gateways.subprocess:subprocess_call(48): executing>> "C:\Users\appveyor\AppData\Local\Temp\1\45cf ������\python.exe" -Wi -m py_compile -
INFO conda.gateways.disk.create:compile_multiple_pyc(367): pyc file failed to compile successfully
python_exe_full_path: C:\Users\appveyor\AppData\Local\Temp\1\45cf ������\python.exe
py_full_path: C:\Users\appveyor\AppData\Local\Temp\1\45cf ������\Lib\site-packages\itsdangerous.py
pyc_full_path: C:\Users\appveyor\AppData\Local\Temp\1\45cf ������\Lib\site-packages\__pycache__\itsdangerous.cpython-37.pyc
compile rc: 1
compile stdout: 
compile stderr: [Errno 2] No such file or directory: 'C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\45cf ������\\Lib\\site-packages\\itsdangerous.py'

The order of the actions may need to be specified so that the pyc compilation always occurs after the linking.

Ordering actions seems wise. Please go ahead with that idea.

I think the unicode characters in the prefix name and path were causing issues on Windows. Hopefully escaping them and using the proper linesep fixes the issue.

You can call compile with an argument for each filename too AFAIK.

The filesystem encoding may be different than the encoding used for stdin. The documentation for sys.stdin, which is what py_compile is reading from, indicates that when the stream is not interactive, stdin uses a "ANSI code page" encoding. I think this can be determined from locale.getpreferredencoding(), but I'd preferred to interrogate the interpreter itself to determine the encoding.

You can call compile with an argument for each filename too AFAIK.

You need to limit the length of the arguments in this case. e96463c implemented this by calling py_compile multiple times and is a good option if the correct encoding for stdin cannot be determined reliably.

failing on spaces. Trying to figure out proper quoting.

The scheme used in b3cb857 will fail to compile .py files if the paths use characters which cannot be represented in the stdin character encoding. For example on windows a path may have UTF-8 characters which are not present in the cp1252 code page. This would be very unusual and the old scheme may have also failed in these cases.

Using compileall has the potential to compile a lot of unnecessary python files which will then be removed. For a large environment this would happen for each noarch package installed. Is this a performance concern?

You can call compile with a list of files as arguments, just remove the - from Jonathan's command line and add the files..

Yeah, that's why I put in that search for extra pyc files. I have no idea what the performance implication is without some sort of benchmark. It's pointed solely at site-packages, which I think dramatically limits potential problems.

Ray, I know that you can compile with a list of arguments. The trouble is that that list of arguments is subject to this hell of encoding nonsense. I'm trying to find ways to skirt that hell.

How about emit to a file as utf-8 a line per file, then write a simple script that reads that back in?

How about emit to a file as utf-8 a line per file, then write a simple script that reads that back in?

compileall can take a file containing a list of file as input via the -i argument. The question is where to store said file?

doesn't matter - some tempdir?

I've heard of pair programming but this is even better! Even if I'm just in the backseat.

Sorry @jjhelmus I'm done for the day on this. I think we're back where we started, but hopefully we know a bit more now.

Hi there, thank you for your contribution to Conda!

This pull request has been automatically locked since it has not had recent activity after it was closed.

Please open a new issue or pull request if needed.