Py_TRACE_REFS is not compatible with Py_LIMITED_API

[vstinner]

Bug report

Checklist

• I am confident this is a bug in CPython, not a bug in a third-party project
• I have searched the CPython issue tracker,
  and am confident this bug has not been reported before

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

No response

A clear and concise description of the bug:

Trying to build a C extension targetting the limited C API (defining Py_LIMITED_API) fails if Python is configured with ./configure --with-trace-refs. Example with PR #108573:

In file included from ./Include/Python.h:44,
                 from ./Modules/_stat.c:16:
./Include/object.h:62:4: error: #error Py_LIMITED_API is incompatible with Py_TRACE_REFS
   62 | #  error Py_LIMITED_API is incompatible with Py_TRACE_REFS
      |    ^~~~~
make: *** [Makefile:3194: Modules/_stat.o] Error 1

The #error comes from Include/object.h:

#if defined(Py_LIMITED_API) && defined(Py_TRACE_REFS)
#  error Py_LIMITED_API is incompatible with Py_TRACE_REFS
#endif

The problem is that the PyObject ABI is different: Py_TRACE_REFS adds two members to PyObject structure:

    PyObject *_ob_next;
    PyObject *_ob_prev;

One solution to make Py_TRACE_REFS compatible with Py_LIMITED_API would be to not add these two members to PyObject, but store the double-linked list outside PyObject. Store it in a different structure, so PyObject stays ABI compatible.

Linked PRs

• gh-108634: Py_TRACE_REFS uses a hash table #108663
• gh-108634: PyInterpreterState_New() no longer calls Py_FatalError() #108748

[erlend-aasland]

See also #63083 and #73104