"dyld: Library not loaded: /lib/libpython3.13.dylib" in 3.13.0a4 MacOS non-framework build with custom prefix

[native-api]

Bug report

Bug description:

The above error was discovered in Pyenv CI check (logs are on the link).

See pyenv/pyenv#2903 (comment) for diagnostics and probable cause.

CPython versions tested on:

3.13

Operating systems tested on:

macOS

Linked PRs

• gh-115737: Correct the install name for non-framework macOS libPython builds. #115750

[ronaldoussoren]

How can we reproduce this? In particular, what is the command-line for configure and any additional customization of the build process?

[ned-deily]

From a quick look at the linked pyenv issue, it looks like this may be related to the changes for iOS frameworks builds (gh-114099). @freakboy3742?

[ronaldoussoren]

I'd still like to know how to reproduce this without having to hunt through the pyenv GitHub. The linked issue seems to use -rpath somehow, which is not something we explicitly support.

[freakboy3742]

@ned-deily As with @ronaldoussoren, I'm trying to work out what the reproduction instructions are... I'll dig in and see if I can reverse engineer the problem.

[native-api]

Attached a Bash trace log from the Pyenv invocation which contains configure/make calls and ennvar assignments. That's what I usually use to diagnose Pyenv issues.
out.txt
If you need a more distilled view, I'll be able to compose it when I return from work in about 10 hours.

Setting rpaths is required in MacOS to find dylibs outside of the default search path -- in Homebrew and under the custom installation prefix.

[freakboy3742]

@native-api Thanks for that detail - based on that, I've been able work out the reproduction case is:

./configure --prefix=$(pwd)/root --enable-shared --libdir=$(pwd)/root/lib && make -j4 all install && $(pwd)/root/bin/python3.13 -m test.pythoninfo

[freakboy3742]

FWIW: it looks like the problem slipped in because while I did test non-framework builds, and non-framework builds in a non-default location, I didn't check a shared (i.e. ,--enable-shared) non-framework, non-default location build.

[ned-deily]

shared (i.e. ,--enable-shared) non-framework, non-default location build

Sorry, I should have thought to suggest testing that one as well originally. I generally avoid --enable-shared builds on macOS as they often lead to problems like this but they are used.

[native-api]

We have enabled --enable-shared by default by user requests. The rationale was it is required for products that link against libpython and it's the setting in official CPython Docker images.

[ned-deily]

To be sure, there's nothing inherently bad about using --enable-shared on macOS, especially when running from an installed Python. You just need to be vigilant during development and testing when running from a Python build directory to make sure the interpreter is not using the old installed shared library rather than the newly built one.

I've commented on the PR on how we can simplify the proposed fix.

That said, I stumbled across another potential problem here with --enable-shared. In the reproduction case above, there is:

--prefix=$(pwd)/root --enable-shared --libdir=$(pwd)/root/lib

Now it turns out that specifying that value for libdir is a no-op since the default value of libdir is $prefix/lib so there's no need to use libdir in this case. However, if you were to use a different value for libdir, then the macOS install_name, i.e. the path to the shared library that is baked into the python executable by the linker, is hardwired to be $prefix/lib so attempting to execute python will immediately segfault. It would be a simple change here to fix the install_name for the enable-shared case to use libdir, which is where the shared library is installed, and, while that does prevent the segfault, the installed Python is still not very usable as the standard library extension modules are also installed into libdir but the default sys.path doesn't take that into account so imports of them fail. This isn't a new issue it seems and has probably been around for almost forever but, as far as I recall, no one has every complained about it so it is probably best to leave well enough alone and not attempt to change this behavior here.