free-threading: PyImport_AddModuleRef race condition

[crusaderky]

Bug report

Bug description:

On a NoGIL interpreter, if two threads call PyImport_AddModuleRef for the same module name at the same time, and that module does not exist yet, it is possible that the two calls to the function return two different objects, only one of which is stored in sys.modules.

The race condition is actually in the helper function import_add_module:

cpython/Python/import.c

Lines 319 to 335 in a50822f

	PyObject *m;
	if (PyMapping_GetOptionalItem(modules, name, &m) < 0) {
	return NULL;
	}
	if (m != NULL && PyModule_Check(m)) {
	return m;
	}
	Py_XDECREF(m);
	m = PyModule_NewObject(name);
	if (m == NULL)
	return NULL;
	if (PyObject_SetItem(modules, name, m) != 0) {
	Py_DECREF(m);
	return NULL;
	}
	return m;

if both threads call PyMapping_GetOptionalItem before either has called PyObject_SetItem, you will get two different modules.

Context

Each SWIG module calls PyImport_AddModuleRef to retrieve or initialise a shared global state module with the hardcoded name swig_runtime_data5, and then stores its pointer in a module-local global variable.

SWIG is actually immune from this race condition because the call to PyImport_AddModuleRef always happens inside SWIG_init, which is an alias for PyInit_<modulename> which holds the GIL.

This race condition only affects lazy initialisation patterns, where PyImport_AddModuleRef is executed for the first time after module init.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

• gh-137422: Fix race condition in PyImport_AddModuleRef #141822
• [3.14] gh-137422: Fix race condition in PyImport_AddModuleRef (gh-141822) #141830

[serhiy-storchaka]

If sys.modules is a dict, we can use atomic PyDict_SetDefaultRef().

    if (PyDict_CheckExact(modules)) {
        PyObject *result;
        (void) PyDict_SetDefaultRef(modules, name, m, &result);
        Py_DECREF(m);
        return result;
    }

But there is no such API for general mapping and there cannot be. So we will need to use some external locking. Or just make sys.modules always a dict, I don't think there are still cases for a non-dict.

[kumaraditya303]

Looking at the code, I think interp->modules can only ever be a dict, if the user code assigns sys.modules then that doesn't affect the interp->modules so we can make this thread safe by relying on the fact that it is always a dict.

I verified it by adding the following assertion and all tests passed:

diff --git a/Python/import.c b/Python/import.c
index 73b94d0dd2a..2d477bab8e1 100644
--- a/Python/import.c
+++ b/Python/import.c
@@ -199,6 +199,7 @@ get_modules_dict(PyThreadState *tstate, bool fatal)
                          "unable to get sys.modules");
         return NULL;
     }
+    assert(PyDict_CheckExact(modules));
     return modules;
 }
 

[colesbury]

Looking at the code, I think interp->modules can only ever be a dict, if the user code assigns sys.modules then that doesn't affect the interp->modules so we can make this thread safe by relying on the fact that it is always a dict.

Yeah, I think that's right, even though it's confusing. It seems like it's been that way since PyImport_AddModule was added in 1.5.

It's kind of weird that if you change sys.modules you affect the Python importlib code path, but not the Python import.c code paths.