Python Finalization¶

Issues with clearing memory at exit¶

Since Python 3.6, a big refactoring started in CPython to clear more and more variables and states at Python exit: in Py_FinalizeEx(), but also in Python main() (for variables which cannot be cleared in Py_FinalizeEx().

Sadly, it caused some nasty issues:

Weird GC behavior during Python finalization¶

When an extension module is converted to the multiphase initialization API (PEP 489), sometimes tests using subinterpreters start to leak.

2020-11-03: bpo-41796: Make _ast module state per interpreter: test_ast leaks.
- FIX: Call _PyAST_Fini() earlier (commit).
  
  Python types contain a reference to themselves in in their PyTypeObject.tp_mro member. _PyAST_Fini() must called before the last GC collection to destroy AST types.
  
  _PyInterpreterState_Clear() now calls _PyAST_Fini(). It now also calls _PyWarnings_Fini() on subinterpeters, not only on the main interpreter.
  
  Add an assertion in AST init_types() to ensure that the _ast module is no longer used after _PyAST_Fini() has been called.
2020-03-24: bpo-40050: Port _weakref to multiphase init: test_importlib started to leak.
- FIX/WORKAROUND: remove unused _weakref (and _thread) import in importlib._bootstrap_external (commit)
2020-04-02: bpo-40149: Convert _abc module to use PyType_FromSpec(): test_threading leaks.
- WORKAROUND 1 (not merged): add a second _PyGC_CollectNoFail() call in finalize_interp_clear().
- FIX 1: Implement traverse in _abc._abc_data (commit)
- WORKAROUND 2 (not merged): add Py_VISIT(Py_TYPE(self)); in abc_data_traverse().
- Regression caused by bpo-35810: Object Initialization does not incref Heap-allocated Types?
- bpo-40217: The garbage collector doesn’t take in account that objects of heap allocated types hold a strong reference to their type
- FIX 2 (bpo-40217): inject a magic function to visit the heap type in traverse functions (commit)
- FIX 3 (bpo-40217): Revert FIX 2 and traverse functions must explicitly visit their type (commit). abc_data_traverse() now calls Py_VISIT(Py_TYPE(self)).
2019-11-22: bpo-36854: Make GC module state per-interpreter: test_atexit started to leak.
- FIX: Fix refleak in PyInit__testcapi()
- WORKAROUND: clear manually the interpreter codecs attributes (search path, search cache, error registry)
- commit

Reorder Python finalization¶

bpo-43687: use unicode_state empty string before unicode_init. without define WITH_DOC_STRINGS
- 2021-04-02: Py_Initialize() creates singletons earlier
bpo-41796: Make _ast module state per interpreter
- 2020-11-03: Call _PyAST_Fini() earlier (commit).
  
  _PyInterpreterState_Clear() now calls _PyAST_Fini().
bpo-42208: Using logging or warnings during Python finalization does crash Python
- 2020-10-30: Call GC collect earlier in PyInterpreterState_Clear() (commit).
  
  The last GC collection is now done before clearing builtins and sys dictionaries. Add also assertions to ensure that gc.collect() is no longer called after _PyGC_Fini().
bpo-40887: Free lists are still used after being finalized (cleared)
- 2020-06-08: Fix finalize_interp_clear() for free lists (commit).
  
  Reorganize code to ensure that free lists are cleared in the right order. Call _PyWarnings_Fini() before _PyList_Fini().
bpo-19466: Clear state of threads earlier in Python shutdown
- 2020-03-09: Py_Finalize() clears daemon threads earlier (commit).
  
  Clear the frames of daemon threads earlier during the Python shutdown to call objects destructors. So “unclosed file” resource warnings are now emitted for daemon threads in a more reliable way.
bpo-39511
- 2020-02-01: PyThreadState_Clear() calls on_delete (commit).
  
  PyThreadState.on_delete is a callback used to notify Python when a thread completes. _thread._set_sentinel() function creates a lock which is released when the thread completes. It sets on_delete callback to the internal release_sentinel() function. This lock is known as Threading._tstate_lock in the threading module.
  
  The release_sentinel() function uses the Python C API. The problem is that on_delete is called late in the Python finalization, when the C API is no longer fully working.
  
  The PyThreadState_Clear() function now calls the PyThreadState.on_delete callback. Previously, that happened in PyThreadState_Delete().
  
  The release_sentinel() function is now called when the C API is still fully working.
bpo-36854: Make GC module state per-interpreter
- 2019-11-20: Clear the current thread later in the Python finalization (commit).
  
  The PyInterpreterState_Delete() function is now responsible to call PyThreadState_Swap(NULL).
  
  The tstate_delete_common() function is now responsible to clear the autoTSSKey thread local storage and it only clears it once the thread state is fully cleared. It allows to still get the current thread from TSS in tstate_delete_common().
bpo-38858: new_interpreter() should reuse more Py_InitializeFromConfig() code
- 2019-11-20: Call _PyExc_Fini() and _PyGC_Fini() later in the finalization (commit).

Release objects at exit¶

Main issue: bpo-1635741.

bpo-1635741: Release Unicode interned strings at exit (commit).

Prevent deadlock in io.BufferedWriter¶

https://bugs.python.org/issue23309

Commit:

commit 25f85d4bd58d86d3e6ce99cb9f270e96bf5ba08f
Author: Antoine Pitrou <solipsis@pitrou.net>
Date:   Mon Apr 13 19:41:47 2015 +0200

    Issue #23309: Avoid a deadlock at shutdown if a daemon thread is aborted
    while it is holding a lock to a buffered I/O object, and the main thread
    tries to use the same I/O object (typically stdout or stderr).  A fatal
    error is emitted instead.

Code:

relax_locking = _Py_IsFinalizing();
Py_BEGIN_ALLOW_THREADS
if (!relax_locking)
    st = PyThread_acquire_lock(self->lock, 1);
else {
    /* When finalizing, we don't want a deadlock to happen with daemon
     * threads abruptly shut down while they owned the lock.
     * Therefore, only wait for a grace period (1 s.).
     * Note that non-daemon threads have already exited here, so this
     * shouldn't affect carefully written threaded I/O code.
     */
    st = PyThread_acquire_lock_timed(self->lock, (PY_TIMEOUT_T)1e6, 0);
}
Py_END_ALLOW_THREADS
if (relax_locking && st != PY_LOCK_ACQUIRED) {
    PyObject *msgobj = PyUnicode_FromFormat(
        "could not acquire lock for %A at interpreter "
        "shutdown, possibly due to daemon threads",
        (PyObject *) self);
    const char *msg = PyUnicode_AsUTF8(msgobj);
    Py_FatalError(msg);
}

Notes¶

To workaround bpo-19565 on Windows, multiprocessing crash at exit, _winapi.Overlapped deallocator leaves the overlapped handle open if Python is exiting, see the commit:

commit 633db6f6a69fd44b4a27e7e216ff7a138f69aaf3
Author: Richard Oudkerk <shibturn@gmail.com>
Date:   Sun Nov 17 13:15:51 2013 +0000

    Issue #19565: Prevent warnings at shutdown about pending overlapped ops.

Python issues¶

2013-10-31: Clear state of threads earlier in Python shutdown. Call _PyThreadState_DeleteExcept(tstate) in Py_Finalize(). This issue introduced corrupted a Python frame of an asyncio daemon thread which leaded to a crash: bpo-20526. I had to revert the _PyThreadState_DeleteExcept(tstate) change.

Cython¶

__dealloc__():

By the time your __dealloc__() method is called, the object may already have been partially destroyed and may not be in a valid state as far as Python is concerned, so you should avoid invoking any Python operations which might touch the object. In particular, don’t call any other methods of the object or do anything which might cause the object to be resurrected. It’s best if you stick to just deallocating C data.

Daemon threads¶

Subinterpreters cannot spawn daemon threads anymore, since Python 3.9: https://bugs.python.org/issue37266
In Python 3.8, daemon threads now exit immediately when they attempt to acquire the GIL, after Py_Finalize() has been called:
- https://bugs.python.org/issue36475 with https://github.com/python/cpython/commit/f781d202a2382731b43bade845a58d28a02e9ea1
- https://bugs.python.org/issue39877 with https://github.com/python/cpython/commit/eb4e2ae2b8486e8ee4249218b95d94a9f0cc513e
change of bpo-19466 caused bpo-20526 regression