“initialization/1 failed for: user:run_all_demos” on Apple Silicon macOS

[sporeking]

Hello,

Thank you for your excellent work!

I am experiencing a persistent issue where the Python library fails to initialize on an Apple Silicon Mac when using a Python environment
managed by Conda.

When I try to run the basic example (scryer-prolog examples/python_demo.pl), it fails with the warning:

❯ scryer-prolog examples/python_demo.pl                                 
=== Basic Python Demo ===
% Warning: initialization/1 failed for: user:run_all_demos
?- 

No Python code is executed.

Scryer-prolog version:

❯ scryer-prolog -v    
cargo:0.10.0

Python version: 3.12.12. I tried to use python@3.12 provided by brew and python@3.12 in miniconda env.

MacOS version: macOS Sequoia 15.3 arm64

I have conducted an extensive debugging process Maybe the root cause is a low-level incompatibility between Scryer Prolog's FFI and the FFI function signatures defined in src/lib/python.pl on this specific platform.

Key Debugging Steps and Findings (Most given by Gemini-cli. I'm not familiar with the implementation).:

1. I confirmed that the libpython3.12.dylib from the Conda environment exists and that its architecture (arm64) correctly matches the Scryer Prolog
   executable's architecture (arm64).
2. Setting the PYTHONHOME environment variable did not resolve the issue.
3. I created a minimal Prolog test that bypassed the library and called use_foreign_module directly with only Py_Initialize and Py_Finalize. This test succeeded, proving that basic FFI calls and Python initialization are possible.
4. Crucially, when I added the signature 'PyRun_SimpleString'([cstr], int) to the minimal test, the use_foreign_module call itself failed immediately.
5. Using nm, I verified that the _PyRun_SimpleString C symbol is correctly exported from the Conda libpython3.12.dylib.
6. I attempted a workaround by commenting out the PyRun_SimpleString signature in src/lib/python.pl and rewriting the predicate that uses it to rely on PyRun_String instead. This did not solve the problem, suggesting that other function signatures in the FFI list are also likely incompatible.

Gemini's Conclusion:
The evidence strongly suggests that one or more of the FFI function signatures in load_python_library_once are incorrect for the arm64 macOS platform, at least when linking against a Conda-provided libpython. The failure of use_foreign_module points to a direct signature mismatch, possibly related to C type sizes or alignment (int, long, i64, ptr) on this platform.

Can you tell me how to solve this or what should I do? Thanks!

[jjtolton]

Wow thanks! The current approach only checks a few common locations. I will make it so that you can specify the location of the shared library. Conda is always a bit tricky but I will put in some instructions for it. I will also add a verbose mode so that it will make working together on issues like this a little easier.

Thanks again!

[jjtolton]

Ok @sporeking this won't completely solve your problem, but I did add the ability to specify your python environment. You can see how it is done here and here for conda.

The only thing is that there might be an upstream issue right now preventing Conda from using all of modules via FFI in Scryer. I'll submit a PR for upstream and make a local fork available until then.

[jjtolton]

I also provided dockerfiles but note again that the Conda one isn't quite working due to the possible issue with upstream Scryer.

[sporeking]

I'm sorry for not making myself clear earlier.
I actually used both python@3.12 from homebrew and python 3.12 env from miniconda.
The python@3.12 from homebrew has the same warning.
And when I used conda, I have specified the location of the Python dynamic library and added the following code:

candidate_python_library('/Users/sporeking/miniconda3/envs/py312/lib/libpython3.12.dylib').

It got the same warning.
So I don't think this is a question about conda itself.

[jjtolton]

Right -- sorry, I skipped a few steps in my explanation.

I think we need to figure out if this is a Mac/Silicon problem or a Conda problem.

I added some options now that give you better control over how the program loads Python, see https://github.com/jjtolton/scryer-prolog-python/blob/main/manual_tests/test_uv.pl#L26-L29.

You just need to specify the path of the python program and the shared library it was compiled with. Try that with your system Python and we can determine if this is a Conda issue or a Mac/Silicon issue. You can also enable the verbose flag for a fuller error trace that I could look at.

[jjtolton]

:- use_module(library(os)).

:- use_module('/path/to/src/lib/python.pl').

:- initialization(main).

main :-
    py_initialize([
        shared_library_path('/opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/lib/libpython3.12.dylib'),
        python_executable('/opt/homebrew/opt/python@3.12/bin/python3.12')
    ]),
    py_run_simple_string("import sys"),
    py_run_simple_string("print(f'Python version: {sys.version}')"),
    py_run_simple_string("print(f'Python prefix: {sys.prefix}')"),
    py_run_simple_string("print(f'Platform: {sys.platform}')"),
    py_run_simple_string("print(f'Architecture: {sys.implementation.name}')"),
    py_run_simple_string("print('macOS arm64 Python 3.12 test: SUCCESS!')"),
    py_finalize,
    halt.

This should be pretty close to the setup you should use

[jjtolton]

I made an upstream patch for Conda. You can use the Dockerfile using my fork to see and example setup in the meantime!

[sporeking]

Thanks for your work! I tried to use the code above to test my system python@3.12:

:- use_module(library(os)).
:- use_module('../src/lib/python.pl').
:- initialization(main).

main :-
    py_initialize([
        shared_library_path('/opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/lib/libpython3.12.dylib'),
        python_executable('/opt/homebrew/opt/python@3.12/bin/python3.12')
    ]),
    py_run_simple_string("import sys"),
    py_run_simple_string("print(f'Python version: {sys.version}')"),
    py_run_simple_string("print(f'Python prefix: {sys.prefix}')"),
    py_run_simple_string("print(f'Platform: {sys.platform}')"),
    py_run_simple_string("print(f'Architecture: {sys.implementation.name}')"),
    py_run_simple_string("print('macOS arm64 Python 3.12 test: SUCCESS!')"),
    py_finalize,
    halt.

But I got a new error:

❯ scryer-prolog manual_tests/test_brew.pl                                                                                                           [14:03:07]
   error(domain_error(py_initialize_option,python_executable('/opt/homebrew/opt/python@3.12/bin/python3.12')),process_init_options/1).
?- 

The dylib is right:

❯ file  /opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/lib/libpython3.12.dylib                                             [14:02:50]
/opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/lib/libpython3.12.dylib: Mach-O 64-bit dynamically linked shared library arm64

Btw, when I use other test scripts, I got the same error: error(domain_error(py_initialize_option,python_executable('xxxxxxxxx')),process_init_options/1).

What should I do next?

[jjtolton]

Yikes, ok thanks for discovering this. I think this is a Silicon incompatability issue. I will adjust the pointers so they work for both systems, but I won't be able to get to it until tomorrow. In the meantime please experiment with the docker! It will help you figure out your app configuration after I make the crate cross platform compatible.

[jjtolton]

@sporeking I am setting up some CI testing for macos so hopefully we can get this resolved soon.

[sporeking]

Thanks for your work!
Here are some results of docker:

Test platforms:

1. Archlinux x86_64 (Linux 6.17.7-arch1-1)
2. macOS Sequoia 15.3 arm64

I have test the docker in both platforms, but get the same error:

❯ docker run --rm scryer-python-demo                                                                 [01:31:00]
   error(existence_error(procedure,append/3),append/3).

For other tests (hello world, python math, http request), docker on archlinux all succeeded, but docker on Mac failed.

Other tests On Mac:
Hello world:

❯ docker run --rm -it scryer-python-1 scryer-prolog -g "use_module('src/lib/python'), py_initialize([shared_library_path('/root/.local/share/uv/python/cpython-3.11.14-linux-x86_64-gnu/lib/libpython3.11.so'), python_executable('/app/.venv/bin/python3')]), py_run_simple_string(\"print('Hello from Python!')\"), py_finalize, halt"
use_module(src/lib/python),py_initialize([shared_library_path(/root/.local/share/uv/python/cpython-3.11.14-linux-x86_64-gnu/lib/libpython3.11.so),python_executable(/app/.venv/bin/python3)]),py_run_simple_string("print('Hello from Python!')"),py_finalize,halt causes: error(domain_error(py_initialize_option,python_executable(/app/.venv/bin/python3)),process_init_options/1)
?- 

Python math and http request are mostly same.

For more tests about docker-conda, I will do tomorrow.

[sporeking]

Another question:
I tried to run the examples/python_demo.pl on my archlinux, and it failed:

❯ scryer-prolog examples/python_demo.pl                                                                                                        [00:36:29]
=== Basic Python Demo ===
1. Simple arithmetic:
   error(type_error(list,'result = 2 + 2'),must_be/2).

I found that the 'result = 2 + 2' is not the list type, and I found that in the Commit c6fbe50, you modified must_be(atom, Code) to must_be(list, Code) (at line 321, the py_run_simple_string).
So I tried to checkout to Commit 54e7ec7, then I got a successful result. (all succeeded except numpy test)

Maybe this is a preliminary change to solve the issue#2?

[triska]

I think must_be(list, ...) should now be must_be(chars, ...), could you please try to use a Prolog string for the code, i.e., "..."? Thank you a lot!

See also #2.