Optimize the traceback Module For Core Runtime Use (and Support Custom Builds Better)

[ericsnowcurrently]

Feature or enhancement

Since gh-110702 (Oct. 2023), we now rely on the traceback module for the operation of the core runtime. The default sys.excepthook() implementation, _PyErr_Display(), imports the traceback module and calls traceback._print_exception_bltin(). If there are any problems it falls back to a C implementation. (I haven't looked closely but, given how much code was removed in that PR, I'm guessing the fallback code doesn't do nearly as much as _PyErr_Display() used to.)

Here's the thing that caught my attention: we still import traceback from disk. That means there are some extra factors we must consider for the module, as a component of the core runtime (effectively):

• importing traceback from disk is "slow" (at least the initial import)
• the module has a lot in it we don't need for sys.excepthook(), which adds to import time (and surface area for possible failures)
• the module has a number of (module) dependencies that might not be necessary for what sys.excepthook() needs
• CPython embedders/customizers would not know to possibly accommodate the module (and its dependencies), whether they remove modules, zip up the stdlib, or something else where the traceback module might be impacted
• the more we can reduce the likelihood of the fallback case, the better, for reasons of performance and complexity (and feature parity?)

(Regarding the points about code-we-don't-need and dependencies, this is similar to what happened with collections.abc, from which we ended up factoring out _collections_abc, to speed up site.py.)

The consequences of the above aren't necessarily¹ especially severe because:

• the default sys.excepthook() (_PyErr_Display()) falls back to a (less informative?) C implementation if traceback can't be imported or there's any other failure related to calling traceback._print_exception_bltin()
• in the vast majority of cases where sys.excepthook() is invoked (i.e. implicitly, via Py_Main()), it happens in response to an uncaught exception right before the process exits

With all that in mind, I think it would be worth doing the following:

• determine if it makes sense to factor out a module (e.g. _traceback_core) containing only the code we need for sys.excepthook() and with minimal dependencies
• if so, do the work
• either way, freeze the module (and all its module dependencies, if any)

CC @pablogsal

Footnotes

1. I anticipate any work done for this issue will not require a significant effort, and the risk of negative consequences if we do nothing is not insignificant, so ISTM it would be worth doing to reduce that risk, however hypothetical it might be. ↩