Skip to content

[BUG]: is support import numpy in subInterpreter (c++ call python moudule) #5909

@yuetianle

Description

@yuetianle

Required prerequisites

What version (or hash if on master) of pybind11 are you using?

3.0.1

Problem description

  1. i want use subinterpreter in multithread

CPython version 3.14
pybind11 3.0.1

this is my c++ code
`
PYBIND11_EMBEDDED_MODULE(printer, m, py::multiple_interpreters::per_interpreter_gil()) {
m.def("which", [](const std::string& when) {
std::cout << when << "; Current Interpreter is "
<< py::subinterpreter::current().id()
<< std::endl;
});

}
void worker(int id)
{
try {
// 在 worker 线程内 创建子解释器(必须在该线程销毁)
py::subinterpreter sub = py::subinterpreter::create();

    // 在激活子解释器并持有对应 GIL 的作用域内运行所有 Python 调用
    {
        py::subinterpreter_scoped_activate activate(sub);
        py::gil_scoped_acquire gil; // 必须持有 GIL 才能安全调用 Python

        try {
            // 直接 import 并使用模块 —— 不要把 module 对象带出此作用域
            py::module_::import("printer").attr("which")(
                std::string("worker ") + std::to_string(id) + " : activated subinterpreter"
            );
            py::module::import("numpy");
            printf("========current id:%d", id);

			py::exec(R"(
			import sys
			print("=== sys.path ===")
			for p in sys.path:
				print(p)
			)");
            // 再做一次示例调用(同在激活+GIL作用域内)
            py::module_::import("printer").attr("which")(
                std::string("worker ") + std::to_string(id) + " : second call"
            );

            // 演示抛出 Python 异常并安全捕获(可选)
            // try {
            //     py::module_::import("nonexistent_module"); // 会抛 error_already_set
            // } catch (const py::error_already_set &e) {
            //     std::string msg = e.what(); // safe: 在 GIL + sub interpreter scope
            //     std::cerr << "Caught python error in worker " << id << ": " << msg << '\n';
            // }
        }
        catch (const py::error_already_set &e) {
            // 这里仍然在 sub interpreter + GIL 的作用域内,所以可以安全地调用 e.what()
            std::string msg = e.what();
            std::cerr << "Python exception inside worker " << id << ": " << msg << std::endl;
            // 将 Python 异常信息转成纯 C++ 异常或返回码
        }
        // gil 离开作用域后不能再操作任何 Python 对象
    }

    // sub 在离开作用域后,在同一线程析构 —— 满足 Python 3.12 的 destroy-on-creator-thread 要求
}
catch (const std::exception &ex) {
    std::cerr << "Worker " << id << " std::exception: " << ex.what() << std::endl;
}

}
int test_main_thread()
{
py::scoped_interpreter guard{};
// 在主线程里可以做一些主解释器的调用(如果需要)
{
py::gil_scoped_acquire gil;
py::module_::import("printer").attr("which")("main: before starting worker threads");
}
py::gil_scoped_release release;

std::thread t1(worker, 1);
std::thread t2(worker, 2);


t1.join();
t2.join();

}`

printer can excute but numpy give me such tips
Python exception inside worker 1: ModuleNotFoundError: No module named 'numpy'
Python exception inside worker 2: ModuleNotFoundError: No module named 'numpy'

when i use these code

py::exec(R"(
import sys
sys.path.append(r"C:\Users\shida\AppData\Local\Programs\Python\Python312\Lib")
sys.path.append(r"C:\Users\shida\AppData\Local\Programs\Python\Python312\Lib\site-packages")
)");

append the numpy lib in sys.path give me can't use in source directory

Reproducible example code


Is this a regression? Put the last known working version here if it is.

Not a regression

Metadata

Metadata

Assignees

No one assigned

    Labels

    triageNew bug, unverified

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions