test_code's test for PyUnstable_Eval_RequestCodeExtraIndex corrupts interpreter state #141536
Description
Crash report
What happened?
Running test_code and then test_capi in the same process crashes the interpreter, either during discovery of the next test after test_capi, or during shutdown (when not running any other tests):
% ./python -m test test_code test_capi test_matters_not
Using random seed: 2744015225
0:00:00 load avg: 0.06 Run 3 tests sequentially in a single process
0:00:00 load avg: 0.06 [1/3] test_code
0:00:00 load avg: 0.06 [1/3] test_code passed
0:00:00 load avg: 0.06 [2/3] test_capi
0:00:06 load avg: 0.13 [2/3] test_capi passed
0:00:06 load avg: 0.13 [3/3] test_matters_not
Fatal Python error: Segmentation fault
Current thread 0x00007f66c3d36780 [python] (most recent call first):
Garbage-collecting
File "Lib/test/support/__init__.py", line 847 in gc_collect
File "Lib/test/libregrtest/single.py", line 206 in _runtest_env_changed_exc
File "Lib/test/libregrtest/single.py", line 319 in _runtest
File "Lib/test/libregrtest/single.py", line 348 in run_single_test
File "Lib/test/libregrtest/main.py", line 393 in run_test
File "Lib/test/libregrtest/main.py", line 423 in run_tests_sequentially
File "Lib/test/libregrtest/main.py", line 563 in _run_tests
File "Lib/test/libregrtest/main.py", line 598 in run_tests
File "Lib/test/libregrtest/main.py", line 779 in main
File "Lib/test/libregrtest/main.py", line 787 in main
File "Lib/test/__main__.py", line 2 in <module>
File "Lib/runpy.py", line 88 in _run_code
File "Lib/runpy.py", line 198 in _run_module_as_main
Current thread's C stack trace (most recent call first):
Binary file "./python", at _Py_DumpStack+0x33 [0x5609715f4474]
Binary file "./python", at +0x323ce0 [0x560971607ce0]
Binary file "./python", at +0x324197 [0x560971608197]
Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x3fdf0 [0x7f66c3d78df0]
Binary file "build/lib.linux-x86_64-3.14/_ctypes.cpython-314d-x86_64-linux-gnu.so", at +0x12a80 [0x7f66c2c36a80]
Binary file "build/lib.linux-x86_64-3.14/_ctypes.cpython-314d-x86_64-linux-gnu.so", at +0x12b0e [0x7f66c2c36b0e]
Binary file "build/lib.linux-x86_64-3.14/_ctypes.cpython-314d-x86_64-linux-gnu.so", at +0x132f2 [0x7f66c2c372f2]
Binary file "/lib/x86_64-linux-gnu/libffi.so.8", at +0x74a0 [0x7f66c2c1e4a0]
Binary file "/lib/x86_64-linux-gnu/libffi.so.8", at +0x78f0 [0x7f66c2c1e8f0]
Binary file "./python", at +0x123cfe [0x560971407cfe]
Binary file "./python", at _Py_Dealloc+0xad [0x56097145cb52]
Binary file "./python", at +0x1462df [0x56097142a2df]
Binary file "./python", at +0x147ab2 [0x56097142bab2]
Binary file "./python", at _Py_Dealloc+0xad [0x56097145cb52]
Binary file "./python", at +0x120e3c [0x560971404e3c]
Binary file "./python", at +0x12118a [0x56097140518a]
Binary file "./python", at _Py_Dealloc+0xad [0x56097145cb52]
Binary file "./python", at +0x15fcb8 [0x560971443cb8]
Binary file "./python", at +0x15fcd7 [0x560971443cd7]
Binary file "./python", at +0x15ff27 [0x560971443f27]
Binary file "./python", at +0x168146 [0x56097144c146]
Binary file "./python", at PyDict_Clear+0x9 [0x56097144c244]
Binary file "./python", at +0x1ac74b [0x56097149074b]
Binary file "./python", at +0x2b4843 [0x560971598843]
Binary file "./python", at +0x2b4b6f [0x560971598b6f]
Binary file "./python", at +0x2b5251 [0x560971599251]
Binary file "./python", at +0x2b5bd6 [0x560971599bd6]
Binary file "./python", at +0x321ec9 [0x560971605ec9]
Binary file "./python", at +0x321f8a [0x560971605f8a]
Binary file "./python", at _PyEval_EvalFrameDefault+0x6bab [0x560971543f78]
Binary file "./python", at +0x27c8ca [0x5609715608ca]
Binary file "./python", at +0x27ca68 [0x560971560a68]
<truncated rest of calls>
Extension modules: _testinternalcapi, _testcapi, _testlimitedcapi, _testmultiphase, _testsinglephase, _testbuffer (total: 6)
Segmentation fault ./python -m test test_code test_capi test_matters_not
The crash happens because test_code creates a Python function it registers as a freefunc with PyUnstable_Eval_RequestCodeExtraIndex, in a module global. After the tests are run, regrtest clears the module, destroying the function, but the registered freefunc is not unregistered or replaced, leaving a dangling pointer. It doesn't show up as a crash until something creates a code object with extras, however, such as test_capi.test_misc, which calls Modules/_testcapi/test_misc.c:test_code_extra, and then those objects are destroyed. I think this has been a lingering test issue for a long time; test_code has had this dubious behaviour at least since Python 3.9, although I think other tests haven't exercised code.co_extra's until test_capi.test_misc was added in 3.11. It's not shown up as a regular problem before because our sequential test runs usually run tests in alphabetical order, and test_capi < test_code.
Given that test_code does all this through the use of ctypes, I think the sensible fix here is to store the Python freefunc somewhere interpreter-specific, like the sys module, that regrtest won't clear. (The interpreter state dict would be even more appropriate, but I don't we have easy access to that from Python code.) Should we ever add ways to unregister the co_extras freefuncs we can make it do that, instead.
CPython versions tested on:
3.14
Operating systems tested on:
No response
Output from running 'python -VV' on the command line:
No response