Get ctypes working

[tiran]

• build libffi from https://github.com/hoodmane/libffi-emscripten
• cp /opt/libffi-emscripten/lib/libffi.a /emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/pic/
• cp /opt/libffi-emscripten/include/ffi* /emsdk/upstream/emscripten/cache/sysroot/include/
• Build with ./build-python-emscripten-node.sh --enable-wasm-dynamic-linking
• Add *shared*\n_ctypes _ctypes/_ctypes.c _ctypes/callbacks.c _ctypes/callproc.c _ctypes/stgdict.c _ctypes/cfield.c -lffi to Module/Setup.local

[pmp-p]

cp /python-wasm/libffi-emscripten/target/lib/libffi.* /emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/pic/
cp /python-wasm/libffi-emscripten/target/include/ffi* /emsdk/upstream/emscripten/cache/sysroot/include/

Maybe that fixture could be avoided with https://bugs.python.org/issue14527 + its pr python/cpython#20451 using pkg-config + setting PKG_CONFIG_PATH

Why not add a specific target libpython.wasm to the makefile built with -s SIDE_MODULE=1 ? that would be --enable-shared
( i think both static and dynamic libpython are usefull, -fPIC everywhere is harmless )

[pmp-p]

just saw python/cpython#32253 and remind of problem i had with ctypes
iirc v8 cannot load synchronously any wasm library with a size >32KiB ( unlike node or firefox), so it would probably need when targeting browser to add --use-preload-plugins somewhere in assets packaging

[hoodmane]

v8 cannot load any wasm library with a size >32KiB

Of course you mean it refuses to synchronously load large wasm libraries.

--use-preload-plugins somewhere in assets packaging

I would encourage people to package their Python side modules as wheels and compile them themselves rather than using preload plugins. This requires more setup code, but it gives much better control and there are is much more mature tooling around wheels and zip files than around emscripten file_packager which is basically write-only and cannot be modified by humans or tools after creation.

[pmp-p]

compile them themselves rather than using preload plugins.

but emscripten_run_preload_plugins and FS.createPreloadedFile have serious limitations. I don't see how you would take out the .so from the wheel and dlopen it synchronously unless you stay in a worker or firefox ( which both bring other serious limitations ).

i just tried with BrowserFS with ZipFS/OverlayFS/InMemoryFS and the only way i've found was to copy file into MEMFS and preload from there (required for image/audio/wasm ), the gap beetween Node and browser is to big there.

[hoodmane]

See here:
https://github.com/pyodide/pyodide/blob/main/src/js/load-package.ts#L227

[pmp-p]

so basically it would be preferable to hook fopen and rewrite the preloading logic instead of trying to rely on emscripten FS ? anyway i guess that would be required to make a wasi shim for the browser.

[hoodmane]

I'm a bit confused about what the filesystem has to do with it? I'm not doing anything crazy like hooking fopen. Emscripten has the function loadWebAssemblyModule which loads a Wasm Module from a buffer. It doesn't do any filesystem operations. You can get the buffer to it however you like.

[pmp-p]

what the filesystem has to do with it?

the low level fonction for preloading audio/images/wasm ( The 3 of them not only 1 like in load-package.ts ) is FS.createPreloadedFile but it does not seem to support handling blob urls ( i tried with various in memory fs ).

if you don't hook the filesytem how would a C extension be able to dlopen ( or load an image take SDL2_image port ) on the fly. "import" is just the tip of the iceberg.

ref: https://emscripten.org/docs/api_reference/Filesystem-API.html#FS.createPreloadedFile

[hoodmane]

Don't use FS.createPreloadedFile, use Module.loadWebAssemblyModule.

[hoodmane]

Right okay, so the point is that Emscripten's dlopen looks in Module.preloadedWasm for the compiled wasm and if it finds it there it doesn't do any filesystem operations. So what you need to do is

const module = await Module.loadWebAssemblyModule(wasmBuffer, {
      loadAsync: true,
      nodelete: true,
      allowUndefined: true,
});
Module.preloadedWasm[dynamic_lib_path] = module;

where dynamic_lib_path is the path that you want to dlopen later.

[pmp-p]

thx @hoodmane
i've found 3 globals ( not in Module but in window , i guess Module is for workers ) that govern preloading : these are preloadedWasm, preloadedImages, preloadedAudios
It allowed me to load directly from a mounted zip by preloading a temp copy in MEMFS, instantiate and then alter the table to set the correct vfs path so C extensions or ctypes/ffi calls can load the files without further assistance.

edit/ it was made from C with one call to https://emscripten.org/docs/api_reference/emscripten.h.html#c.emscripten_run_preload_plugins + a python call to run_script ( eval for js ) to fix the table after the yield. possibly a job for importlib.invalidate_caches()

[hoodmane]

not in Module but in window

Really you should be building with -s MODULARIZE=1 and then you will find these in Module. Not ideal to allow the huge number of Emscripten variables to be dumped into global scope.

[pmp-p]

you should be building with -s MODULARIZE=1

that seems a very good idea !

@tiran that would need to be set at the makefile stage of -o python.html Programs/python.o of browser target ?

[tiran]

MODULARIZE requires changes to our browser-ui. My JS skills are on total noob level. The worker is failing with some messages related to promise and onMessage:

Error: Promised response from onMessage listener went out of scope

[pmp-p]

Error: Promised response from onMessage listener went out of scope

same js level there that sounds scary, i'm wary about worker since they have very choppy input and no audio, and worse i'm quite sure i have read somewhere they do not fit well with dynamic linking ( or was it threading? )