Python Subinterpreters

See also Python Finalization.

TODO list for per-interpreter GIL

Search for Subinterpreters issues at bugs.python.org.

Meta issue: per-interpreter GIL. Issues:

Enhancements:

Limitations

Not supported in subinterpreter:

  • os.fork(): it may be possible to fix it.
  • signal.signal()
  • static types

Current workarounds:

  • Disable GC
  • Disable many caches like frame free list
  • etc.

Convert static type to heap type

See: Convert static types to heap types: use PyType_FromSpec().

Example: Modules/_abcmodule.c.

Decrement the type reference counter in the dealloc function. Something like:

static void
my_dealloc(my_data *self)
{
    (...)
    PyTypeObject *tp = Py_TYPE(self);
    tp->tp_free(self);
    Py_DECREF(tp);
}

Add a module state to a module

Example: Modules/_abcmodule.c.

Add traverse, clear and free functions to the module to better collaborate with the garbage collector. Otherwise, the GC fails to break reference cycles.

Heap allocated types

Modules/_randommodule.c:

PyObject *Random_Type = PyType_FromSpec(&Random_Type_spec);

Example:

$ ./python
Python 3.9.0a6+ (heads/frame_getback:6bde4d96c7, Apr 29 2020, 03:02:24)
>>> import _random as mod1
>>> import sys; del sys.modules['_random']
>>> import _random as mod2
>>> mod2.Random is mod1.Random
False
>>> mod1.Random.x=1
>>> mod2.Random.x
AttributeError: type object '_random.Random' has no attribute 'x'

Multiphase initialization (PEP 489)

See _abc module.

  • PyInit__abc() calls PyModuleDef_Init
  • PyModuleDef has slots, at least Py_mod_exec.

Get module

Create module:

_PyModule_CreateInitialized(struct PyModuleDef* module, int module_api_version)

Members:

  • PyModuleDef.m_base.m_index: int
  • PyInterpreterState.modules_by_index: list

PyModuleDef_Init() assigns an unique index to a PyModuleDef. It is called by _PyModule_CreateInitialized().

_PyImport_FixupExtensionObject() and import_find_extension() call:

_PyState_AddModule(PyThreadState *tstate, PyObject* module, struct PyModuleDef* def)

Modules with slots must not be added to PyInterpreterState.modules_by_index.

Module State

Find a module:

m = PyState_FindModule(&posixmodule);

From a module:

void *state = PyModule_GetState(module);